Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Writing and using php streams and sockets tek11
1.
2. Idea originating in 1950’s
Standard way to get Input and Output
A source or sink of data
Definitions
C – stdin, stderr, stdout
C++ iostream
Who uses them
Perl IO
Python io
Java
C#
3. These are the people you can blame for the PHP streams implementation
4. We will write code (in pairs)
There will be a quiz (with prizes)
You get out what you put in – participate!
This is really 3 talks in one -
Streams
Filters
Sockets
8. Stream
Contexts
Stream Stream
ALL IO
Filter Wrapper
9. Stream
› Resource that exhibits a flow or succession of
data
Wrapper
› Tells a stream how to handle specific
protocols and encodings
Context
› A set of parameters and options to tell a
stream (or filter) how to behave
10. Scheme
› The name of the wrapper to be used. file, http, https, ftp,
etc.
Target
› Depends on the wrapper, filesystem uses a string path
name, ssh2 uses a PHP resource
home/bar/foo.txt
file:///home/bar/foo.txt
http://www.example.com/foo.txt
ftp://user:pass@ftp.example.com/foo.txt
php://filter/read=string.toupper|string.rot13/resource
=http://www.example.com
11.
12. flock
wrapper limitations
non-existent pointers (infinite loops can
and will happen)
error handling
13. Parameters
Options
Modify or enhance a stream
stream_context_set_param
stream_context_set_option
stream_context_create
28. There are no interfaces
Implement as though there were an
interface
Seekable is optional
Flushable is optional
Directory support is optional
29. fopen
file_get_contents
Information
Return true or false
$this->context will have any context
metadata
Code
30. fread
fgets
file_get_contents
Information
etc…
Return string data or false
$this->context will have any context
metadata
Code
31. fwrite
file_put_contents
Information
get in a string of data to deal with
return how many bytes you wrote
Code
32. feof
file_get_contents
fread
Information
etc…
Return true or false
$this->context will have any context
metadata
Code
33. fclose
file_get_contents
Information
Don’t return anything
any cleanup should go here
Code
34. fstat calls stream_stat
EVERYTHING ELSE uses url_stat
Good idea to do both
Return an array of data identical to stat()
38. Create a custom wrapper for any kind of
network or shared memory storage you
want (except s3 which is way overdone)
Implement at least the basics – extra points
for more features
39. Wrapper for wincache (btw, porting this to
apc would be a breeze)
Does basics + seek + flush (since we’re
caching) + rename + unlink
No directories
41. Data in s3
Data locally during development
Easy switch out if alternative storage is
ever desired
Storing image files
42. Existing Zend Framework Code
Register the s3:// wrapper
Use a configuration setting for the stream
to use for all images on the system
43. Store and edit template files in a
database
Have the snappiness of including from
disk
Minimal Configuration
44. db:// stream
simple stream wrapper that looks for the
template in the db, and writes it to the
filesystem before returning the data
The cached location is FIRST in the
include path, so if it fails, the db stream
gets hit
45. Talk to mercurial (hg binary)
hg communicates via command line
continue to pipe additional commands
46. Use proc_open to keep a pipe to the
binary going
Pass commands through stdin pipe as
necessary
Abstract this out to other binaries that
are used by the system
47. 1. Name 3 Built in PHP streams.
2. What is a Context? A Wrapper? A
Stream?
3. How do you identify a stream?
4. Name two extensions that provide
additional PHP streams.
51. Performs operations on stream data
Can be prepended or appended (even
on the fly)
Can be attached to read or write
When a filter is added for read and write,
two instances of the filter are created.
52.
53. Data has an input and output state
When reading in chunks, you may need
to cache in between reads to make
filters useful
Use the right tool for the job
60. onClose
basically a destructor
Information
no return
Code
61. MUST return
› PSFS_PASS_ON
› PSFS_FEED_ME
Information › PSFS_ERR_FATAL
You get buckets of data and do stuff to
them
Code
62.
63. $in and $out are “bucket brigades”
containing opaque “buckets” of data
You can only touch buckets and
brigades with the stream_bucket_*
functions
You get a bucket using
stream_bucket_make_writeable
64. Given a list of bad words (we’ll use the 7
bad words), write a filter to remove them
from text
Given a commonly misspelled word, fix
the spelling in the text
65. I only did the bad words one, and
created an extremely naïve regex that
stripped the “bad” words from whatever
text it finds.
It buffers the entirety of the text before
doing the replacement – this could be
done by looking for word boundaries
and doing it piecemeal to improve
performance
66.
67. 1. What term is used to describe data
handling in filters?
2. Name two built in filters you should be
using and why.
3. What is the default mode filters are
appended with?
4. What is the difference between
appending and prepending a filter?
70. Socket
› Bidirectional network stream that speaks a
protocol
Transport
› Tells a network stream how to communicate
Wrapper
› Tells a stream how to handle specific
protocols and encodings
71. Network Stream, Network Transport,
Socket Transport
Slightly different behavior from a file
stream
Bi-directional data
72. Sockets block
› stream_set_blocking
› stream_set_timeout
› stream_select
feof means “connection_closed”?
huge reads or writes (think 8K)
stream_get_meta_data is READ ONLY
73. New APIS in streams and filesystem
functions are replacements
Extension is old and not really kept up to
date (bit rot)
Extension is more low level
stream_socket_server
stream_socket_client
76. Given two files of quotes, write a server
with PHP streams methods to return
random quotes, optionally allow client to
pick which file to retrieve quotes from
Given a simple tcp server that will return
a random quote – make a client that
pulls quotes for users
77. socket_server.php – uses
stream_socket_server and
stream_accept
socket_client.php – uses
stream_socket_client and a bit of loop
magic to make a little “cli” app
78. 1. How is a socket different from a stream?
2. What transports does PHP provide by
default?
3. Name the two most commonly used
stream socket functions.
4. How can a user add additional
transports to PHP?