Pyrasun EmberIO

EmberIO is a high-performance Java library for managing and performing I/O.

See:
Description

Packages

pyrasun.eio	The pyrasun.eio package is the main package for the EmberIO library, and contains the core EmberIO classes.
pyrasun.eio.handlers	The handlers package contains the EIOEventHandler abstract base class, and all of the default concrete implementations of EIOEventHandler.
pyrasun.eio.helpers	This package defines helper classes which make common EIO tasks easier to accomplish than is the case with the core EIO classes, which are sometimes a bit too generic and daunting to the semi-casual EIO user.
pyrasun.eio.protocols.bytearray	This package contains the bytearray protocol.
pyrasun.eio.protocols.object	An EmberIO protocol used for sending Serialized objects around.
pyrasun.eio.protocols.string	The incredibly stupid and (almost) worthless string protocol.
pyrasun.eio.services	The services package defines various services that EmberIO supports.
pyrasun.eio.services.bytearray
pyrasun.eio.services.generic
pyrasun.eio.services.object
pyrasun.eio.services.telnet
pyrasun.eio.util	Everybody who's anybody has got a 'util' package, and EIO is no exception.

EmberIO is a high-performance Java library for managing and performing I/O. It's main features include:

• Automated NIO buffer management, including support for partial reads and writes in a non-blocking environment
• Management of one or more NIO Selectors for performing multiplexed I/O. This management is invisible to the application code.
• Support for many different threading models, and for blocking and non-blocking I/O.
• Support for both NIO-based I/O using NIO channels, and non-NIO I/O resources such as multicast sockets. Differences between NIO and non-NIO resources are completely hidden from application code by default.
• Different I/O models give different performance characteristics, with each model representing a tradeoff between resource utilization, latency, throughput, and the physical size of data being read or written.

The biggest bang for the buck you'll get out of EmberIO is in the buffer management, built-in support for a variety of threading models, and generally extracting the pain out of using NIO intelligently. In addition, note that while one of the initial motivations behind EmberIO is to make using NIO easier, it is being expanded to handle non-NIO I/O resources as well.

One of the most interesting EmberIO features is the threading model support - it offers a boatload of them, each of which is targetted towards different I/O scenarios. Almost any scenario may be configured to be used in either a blocking or non-blocking manner (with a few exceptions where the model constrains you to one or the other). On top of this, EmberIO supports both traditional blocking Java I/O semantics (which I'll call BIO here - thanks to James Strachan for keying me on this term!) and newer NIO asynchrous/multiplexed/non-blocking semantics (which I'll just call NIO).

A non-exhaustive list of threading models include:

• BLOCKING_ACCEPTOR
  Creates a dedicated thread for handling accept() calls for a socket server in a blocking manner.
  
  
• NONBLOCKING_ACCEPTOR
  Handles socket server accept() calls in the traditional NIO manner e.g. by using the OP_ACCEPT interest bit in a Selector. For most people this is more of a pain in the ass than anything else, and BLOCKING_ACCEPTOR would be more appropriate.
  
  
• USE_RWP_POOLS
  Uses seperate thread pools for handling READ, WRITE, and PROCESSING events. Each of these will be handled indepedently (at least from a threading perspective). Most useful if you anticipate doing large READs or WRITEs or both. Any of the *_POOL variants aggressively use a PROCESS and WRITE FIFO queue to decouple read/write events from the events themselves and the physical I/O (e.g. READs and WRITEs happen completely asynchronously).
  
  
• USE_ONE_POOL
  Uses a unified thread pool for all events. Often useful if you have small READs and WRITEs and want to reduce latency significantly.
  
  
• DEDICATED_READER
  Sets up EmberIO threading and I/O to be performed in like old-style BIO. Individual sockets (or what have you) are setup with dedicated blocking reader threads, and writes are performed in a blocking manner directly to the I/O resource.
  
  
• SELECTOR_READER
  Performs READ operations within the context of a Selector's main event loop thread. This forces all reads to be done in a non-blocking manner (after all, we don't want to block our selector loop!) and can only be used for NIO-enabled I/O resources (e.g. right now, only Sockets).
  

The above models are configurable per connection type. So, for example, you could configure one type of connection using the DEDICATED_READER strategy for the lowest possible latency, and another type of connection could be configured for USE_RWP_POOLS where many connections are expected with variable I/O sizes, some of them potentially large.

In addition to the various threading models, EmberIO includes a few other features to help the developer tune the model towards the best balance of resource usage/throughput/latency that matches their needs. Most of these are configurable, but a few are hard-coded into the library to work around known NIO (or plain BIO issues) and boost performance.

One thing to keep in mind about EmberIO is that all features are to some extent intertwined, and some features negate the use of others. As one example, use of DEDICATED_READER effectively negates the greedy operations feature, since DEDICATED_READER channels bypass the Selector entirely. Keep this in mind when you're trying out the various EmberIO configruations, and above all use your head!

• Greedy operations.
  EmberIO inherently is pretty aggressive in its use of threads, but at the same time NIO tends to leads toward excessive thread switching. On top of this, Selector performance isn't always what it could be. Because of this, greedy operations were born. Greedy operations basically enable a thread to do "N" units of work once it has control. For example, a READER could be configured with greedy ops set to "20", which would mean that the READER would try to read up to 20 objects before relinquishing control. Judicious use of greedy ops can significantly boost your server's throughput by doing more outside of the Selector, and minimizing use and interruption of the Selector.



• Throttles.
  It's easy I/O to run away out of control in Java applications. This is true in particular when you're using asynchronous READ and WRITE operations. You can end up in a situation where either your I/O threads are eating all of your CPU time, or just as bad, your PROCESS or WRITE queues start filling up monstrously and you start running out of memory. Really, really early implementations of EmberIO didn't address this problem at all, and as result it was common under my stress tests to end up with a WRITE queue containing several hundred thousand objects waiting to be written, for the network to be saturated with a storm of packets, and for PROCESS workers to be starved (or at least seriously undernourished). The end result was a serious unbalanced utilization of available resources, out of memory errors under long lasting heavy loads, and latencies in the tens of seconds. Yuck.
  
  As of 0.3 Alpha, EmberIO supports throttles to address this problem. You can configure EmberIO to throttle READs or WRITEs so that they shut off if either the PROCESS or WRITE queues fill up past a certain threshold.
  
  On the READ side, for most models we turn off READ interest if the PROCESS queue passes this throttle threshold, which effectively means we stop reading. This state persists until a restart threshold is reached - basically, the PROCESS queue is drained sufficiently, at which point we re-inject READ interest. If the DEDICATED_READ model is used, the job is much simpler - we just block the READ thread until the restart threshold is reached.
  
  On the WRITE side, we physically block a thread trying to WRITE in most models if the WRITE queue threshold is exceeded. One gotcha here is that we can't do this if the writer is an EIO thread, since we'd be potentially deadlocking ourselves. To get around this, right now EIO will only block on non-EIO threads (this is done by checking the ThreadGroup of the writing thread). A more sophisticated model may be implemented in the future, since this current one is a bit white-trash and naive. For the DEDICATED_WRITE scenario, we don't do anything special right now - we assume the I/O resource itself will throttle us, and in case no WRITE queue is used, so there's nothing to fill up. This should probably be changed in the future, to avoid flooding the network.
  
  
• Thread Priority
  EmberIO lets you set worker threads' thread priorities along with the pooling strategy you're using.
  
  
• Read and Write Optimizations
  EmberIO has a bunch of optimizations to try to efficiently get data in from your I/O resources and back out again. In particular, we try reads and writes a few times if they are incomplete, even in non-blocking mode. This works extraordinarily well because most JDKs and TCP/IP stacks will often return/write just one byte, even if all the data (or buffer space) you need is there. Quite often a subsequent read/write immediately after will snarf/write the rest of the data.
  
  
• Auto Management of Blocking Semantics
  EmberIO auto-manages the blocking configuration of your channels so you don't have to. More importantly, it does this in a safe manner that avoids the dreaded IllegalBlockingModeException. While you should generally try to use non-blocking I/O for everything, this isn't always a feasible option, and EmberIO makes sure you can write blocking code within it just as easily as you could in the old BIO model.

For more information, see NOTES.txt in the main distribution directory for a more complete outline of what EmberIO's motivation is and what problems its intended to solve. This is an early text version of the blog entry first describing EmberIO in detail. See also DEEP_THOUGHTS.txt, in the same directory, for a much more in-depth explanation of EmberIO - what it does, what the classes are, how to use it, and some exploration into the source code.

Since:

EmberIO 0.3 Alpha