Well, like an idiot, I’ve had this update ready to go for a while (before xmas of 2013) and had left it in a beta state. I never heard anything bad back from the beta testers (which could be good or bad I guess!) but I’ll assume things are working ok.
So it’s probably time to move this over into “prod” and get it out there as I’ve had a few requests recently for this update.
If you need to parse older log files (5.1 or older) you’ll need the standalone parser for that. You can find it linked inside the content here.
For those of you who already have the Log Viewer installed it should auto update – if it does not, or if it breaks somehow, please uninstall it via the Java Control Panel console (or appropriate interface for your chosen OS), and then re-install it from the JNLP file linked above.
Thank you and let me know if you’re having issues with log files! I had a very limited subset to test with when making the update so it’s quite likely there will be errors lurking out there somewhere.
As I am doing some planning for View 5.3 and the new dynamic PCoIP tuning feature I am debating re-writing the PCoIP Config Utility from the ground up. I’d like your feedback on a few items though, the first of which is this-
One “issue” with the utility as it stands today is that it requires Admin privileges to run. This can be dealt with in various ways but there’s only really one reason it requires escalation – the PCoIP registry entries are in HKLM and not HKCU. In order to get access to the keys and, most importantly, make changes to them, you need higher privileges.
This could be gotten around by changing the security on the Registry keys that house the PCoIP settings, but I don’t know if this would be something frowned upon by most users and/or their security teams.
So, I am asking: Are people OK with things as they are and having the app require escalation, or would they prefer to make other changes and have the app run without escalation? If so, what changes are they willing to do?
Also, I am open to other thoughts/options on how to implement this as well. One option I tossed around a while ago was to install a system-level service with access to the required Registry keys and then have the front-end piece talk to that service to apply changes. This way, you could install the service once into a master image and users could run the utility as-is and stop/restart it without any special permissions. Is this a better way?
Please drop your thoughts into the comments.
Dropping in a quick update from VMWorld Europe…
If you read my previous post you know that a new feature was added to View 5.2 that was stated as something along the lines of “GPO options take effect immediately for PCoIP”. Well, it was true but it didn’t exactly work as intended. It worked only with a very small number of options and hence was rendered not terribly useful.
I am happy to say that I have been working with near-release ready code for View 5.3 and this feature now seems to work as intended for pretty much all of the important tuning options. This is great news! It means I have to go back and dust off my work on the real-time tuner tool I had began building at the start of the year and combine it with some updates to the PCoIP Configuration Utility, but I believe I can deliver a really solid workflow for the creation, distribution, and (dynamic) application of PCoIP tuning profiles.
Stay tuned in the coming weeks and I’ll keep you updated as to how things progress.
First off, an apology. This update took way too long to get into your hands, but here it is… In this post, I’ll explain the new update, and then likely dive into far more detail than anyone ever wanted to know about the new parsing framework included in this release.
Update for View 5.2
Goodbye standalone parser
With this release the PCoIP Log Viewer now supports parsing View 5.2 logs. There’s a pretty major change in how this functionality is provided – View 5.2 log files should only be parsed directly from the Viewer. Let me repeat that – the standalone parser will not work for 5.2 log files, you now parse these directly in the Viewer. For any log files prior to View 5.2 you will still need to use the standalone parser. I realize this is likely to cause some confusion and frustration, but I had to make a break from the “legacy” parser if I was going to maintain my sanity. It also sets the stage for some future development I hope to do, and which I’ll talk more about later. This also means, for the moment, if you depend upon embedding the parser into your VDI images to pre-process the log files before opening them in the Viewer (and I know some people have scripted this) that you are out of luck. There’s two ways to look at this – I am the Dark Lord Satan out, specifically, to ruin your life, or, your scripting is now simplified in that you no longer need to pre-process the files or add an executable to the master image/clones. Keep the logic that moves the log files out, and then just open THOSE in the Log Viewer. However, if I get enough hatred and anger, I am certainly open into looking at making a new standalone parser based off the new framework. The catch with that is that it may require Java as a pre-req, or need to be wrapped into a much larger executable, so it’s not a perfect replacement.
Opening View 5.2 log files
With the log file parser now integrated into the Viewer itself, using the new View 5.2 support is as easy as dragging and dropping a “raw” logfile into the Viewer window, or opening the file via the Open Log file menu option. Just to be clear by “raw” logfile, I mean the ‘pcoip_server_<year>_<month>_<day>_<id>.txt (pcoip_server_2013_05_17_00000b90.txt) file. You can still open regular XML files from the standalone parser the same way you always have. Once opened things should look and work more or less as they always have, no new changes to learn there. Here’s a video of the basic usage (except you now CAN open files from the file dialog, and the parser warning is gone):
Getting the Update
All you should need to do is launch the PCoIP Log Viewer while connected to the internet and it should auto-update. If you don’t have the Log Viewer installed already you’ll need Java 6.x or greater installed, then you can go here to get it: PCoIP Log Viewer Webstart
This was a big update, while it doesn’t look like it on the surface the underpinnings of the parser are a major update (as you’ll see below) so I expect there to be bugs. Please let me know what you run into. If you can please include a link to the log file causing the issue. Contact me here (via comments) or on twitter @rexremus and I’ll do my best to get things fixed up. I hope you enjoy this update and get some good use out of the tool. If all you care about is the update – stop here, the rest of this post is going to dive into the details of the new parser and why I wanted to make this change. If that doesn’t sound interesting, get out now while you still can!
New Parsing Framework
The Need for Modular and Extensible Parsing
The previous standalone parser rapidly grew into a mess. Why? Because I had developed an unhealthy desire to have one singular parsing “engine” that could handle anything thrown at it. There are several ways to do this, and the least desirable among them is really what the parser had become over the years – it was a singular parser instance with a ton of exception/quirk-checking to be able to deal with all variants of the PCoIP log files.
This is a trap that many of us fall into. You begin by writing what you expect will be a quick and dirty solution to a problem with no real thought of modularity, flexibility, or maintenance. Then people actually start using it, so now you have a community to support. A minor change needs to be made (like a small change to the content you are parsing) and you just lay it in, no need to refactor things for such a small change, right? Well, one small thing becomes 12, and some of those end up being pretty significant changes that are made worse because of previous changes that required some really hacky workarounds. And next thing you know, you are maintaining a complete mess.
This is where things were at in the standalone parser after the release of View 5.1. That was a major update and required jumping through so many hoops in the existing code I had vowed (to myself anyway) I would not do it again. If I had to change the parser again I was going to build it new, and take the lessons I learned from supporting the original version to make something easier to maintain going forward. I’d given up all hope of the log files staying consistent at this point – and by consistent I don’t mean never adding new data, that’s always to be expected; by consistent, I mean not altering the way existing data is presented and/or ordered so that only new data was required to be dealt with on a new release.
So once I had the time to deal with an update to 5.2 (after hearing from many people that 5.2 logs were not working) I began working out a simple, I hope, to maintain parsing framework.
The New Framework
The basis of the new framework isn’t anything new or exotic, it’s just a few changes that should help make things much more flexible going forward. At a really high level, I just decomposed the monolithic parser into a few component parts as shown below:
Here you can see a parsing component, a log file “model” component, and a formatting/output component. The flow of running a document through here works mostly as it’s shown – the parsing component is responsible for converting the “raw” input into an intermediary container – the PCoIPLogFile object – which can then be fed into a formatting component that takes the intermediary container and “renders” it out to a new form.
All of these steps existed in the monolithic parser, but there was no easy way to extend or change any of the individual pieces. Here, each component represents a unique entity and much better encapsulates the specific functionality it is responsible for. Next we’ll break down some of the individual pieces and talk about how they are implemented
The log parser box in the high-level diagram really represents a base class which can then be extended to create version specific parser instances.
I decided to base the parsing side of the new framework around a SAX-like implementation. If you are unfamiliar with SAX I will give a mangled, tremendously abbreviated breakdown of it here- SAX is used to parse XML documents and uses an event-based model. Rather than trying to load the entire contents of an XML document into memory (that is DOM parsing), SAX defines a set of methods that will be called at various meaningful points as the document is parsed such as startDocument(), endDocument(), and startElement(). These allow you to take action within the context of the event, for example creating your intermediary object in the startDocument() method, and cleaning it up in the endDocument() method. For extracting specific information from XML elements startElement() methods get passed information about the element that triggered the event which you can then examine and determine if it’s an element you care about. SAX is great when you have to parse very large files as it’s much more memory efficient than a DOM parsing implementation, and it gives you flexibility in that you can extend existing classes that implement all required methods (far more than I covered here) and only override the ones you care about – which may be as simple as just implementing the startElement() method.
When creating the new parsing framework I decided to use an abstract base class so that I could provide a handful of methods that I don’t see changing very much, as well as house a few data structures and constants that I knew from prior experience would be used throughout the framework. I could have (maybe should have?) gone with an interface here and then created a base parser class off of that and then have all “real” parser implementations extend that base class, but it seemed like a wash in the end as I was not sure the extra flexibility would be of significant benefit. Building a parser for these log files is a non-trivial exercise, it’s not just 1-2 methods you might tack onto another class and quickly flesh out. So in this case, I went with an abstract base class and handled as much core functionality as I could there.
Looking at a PCoIP log file we see that lines have a basic structure:
06/13/2013, 03:28:01.456> LVL:2 RC: 0 MGMT_SYS :UDP port values: base=4172, range=10, set=4172 06/13/2013, 03:28:01.456> LVL:2 RC: 0 COMMON :load_crypto_lib: FIPS mode enabled=0 06/13/2013, 03:28:01.488> LVL:2 RC: 0 MGMT_SSIG :(tera_xml_init) -- Software Build ID: soft_pcoip_rc_3_12 22.214.171.12451 06/13/2013, 03:28:01.503> LVL:2 RC: 0 COMMON :Initializing Winsock2 06/13/2013, 03:28:01.519> LVL:2 RC: 0 SCNET :PCoIP Soft Crypto Module Build ID: trunk 126.96.36.19958 06/13/2013, 03:28:01.846> LVL:2 RC: 0 MGMT_ENV :Setting ENV variable[ 7]: pcoip.tcp_port = 4172
If we break this down a little we see that every line starts with a timestamp, followed by a logging level, an RC code, a line “type” or category, and then the actual data or content of the line. I decided to model my SAX-like parsing around the line type element. The base parser will run through the file and call specific methods for each line type encountered (COMMON, MGMT_SSIG, SCNET, etc.) as well as methods for the file start and end. Each of these methods are defined as abstract in the base parser class and must be implemented in any concrete classes.
I then created a default implementation of a parser (DefaultPCoIPLogParser) that is based around parsing a View 5.2 log file. In this class I provide implementations of the methods for every line of “interest” in the log file needed to display it properly in the Log Viewer. This base implementation can then be extended when the log file format changes again, and only the methods that deal with the lines (really, the line types) that changed will need to be updated. This greatly simplifies dealing with new formats because there’s no need to worry about backwards compatibility as there was in the monolithic parser – you can do whatever hacky, ultra-version-specific parsing tricks that make dealing with the current log file as easy as possible without any concern of breaking the parsing of a previous version and then let the existing methods of the parent class deal with everything else that didn’t change. In fact, if the inherited method doesn’t blow up or cause any other “damage” to the data placed in the intermediary object, you can call it from your overridden method, and then come back and “correct” or populate only the data pieces affected by the new log format – this way you don’t need to copy/paste or otherwise replicate the existing parsing from the parent class and it saves you more time when creating the new parser version.
This is an intermediary object used to store the important data parsed from the log file. It’s purpose it’s really just to be a bridge between the input and output of the parsing process – a parser generates a log file object, and a formatter uses it as input.
This isn’t to say the log file object is just a dumb container. It does contain a few methods, the most important of which is the validate() method. Because of the move to a SAX-like parsing front-end it became harder to determine if a log file was “good” or “bad” from within the parser alone. Certainly things like an individual line can be validated, but does one missing line, or one malformed entry mean the entire file is useless? Probably not, but it might, it just depends on which line it is and possibly the context around it. Rather than trying to put that kind of logic into what essentially amount to “stateless” methods within the parser (or going through a lot of effort to give them state) it made a lot of sense to just encapsulate the notion of “validity” into the object storing the entire document state itself. Once the parsing is done, the parsing workflow validates the log file object before passing it to a formatter. If the file is invalid, an exception is thrown and can be dealt with. This greatly simplifies error checking in that parser methods need only check for validity of a specific line (as that’s all the context they can be guaranteed to have) and the log file object itself can check for overall validity given that it can assume the point where the validate() method is being called, it should be considered a “complete” representation of a valid file.
Just as the parser box in the high-level diagram represented an extensible component, the same is true for the formatter box. In this case though, I did go with an Interface as there’s only a singular method to be implemented and it’s quite possible to just tack it onto an existing class if the formatting required is not terribly complex. For more complex formatting it’s likely a dedicated class will be built which can then be sub-classed as required to tweak or change behavior.
As you can see there’s just one method – format() – that returns an InputStream containing the formatted output. This allows the resulting formatted content to be pretty easily stored to a physical file, or to be piped around and read from in memory. Java’s streams (and all their variants) are fairly rich and well understood at this point so this seemed a pretty low-level and flexible result to get out of the formatter.
For the sake of the current work, I’ve created an XML formatter to allow for the existing visualization chain to be used in the viewer. But I can easily imagine an Excel based formatter to generate a physical file, or a JSON formatter to allow the parser/viewer to move to a cloud-based application and send data down to a browser or mobile device. All of these should be fairly easy to implement.
Gluing it all together – The Factory
Well, saying “all” might be a bit of a lie, but I wanted an easy way to make sure you always get the “right” parser, despite there only being one that’s of any use right now. So I decided to build a ParserFactory to do that.
Basically, the Factory will scan the classpath looking for PCoIPLogParser implementations and then register them with itself to build a catalog of all available parsers. Within the app, when we need to parse a file we can then call the cleverly named getParser(logFile) method. This method examines the raw log file and extracts the “version” of the file based upon certain elements, it then queries every registered parser for it’s supported log file version. When it finds a match, it returns that parser. Currently there’s not much thought given to conflict resolution, e.g. what if more than one parser can handle a given version of file? But my thought on that is, within any given project, you probably wouldn’t need more than one way to “parse” a file – you might want more than one way to format it, but really you only care about getting it into the PCoIPLogFile object and then working with it from there. So while more than one parser capable of dealing with a View 5.2 log file might exist, it isn’t likely that you’d see them together in the same project where they might cause collisions.
The horrible diagram above shows (poorly) the basic flow of parsing a file:
- Get the file
- Ask the ParserFactory for a parser, passing it that file
- The factory determines the version of the file
- The factory queries all the registered parsers for their ability to parse it
- A valid parser capable of parsing the file is returned from the factory
- The parse() method is invoked on the file and the PCoIPLogFile object is returned
- The PCoIPLogFile is validated (this is where you can bail out before trying to format)
- If the PCoIPLogFile object is valid, it is formatted
- The returned InputStream is used as required – written to disk or passed to another method/process for display or further manipulation.
I might also consider a factory for the formatter classes as well, something where you can pass in a desired format and get the right formatter back – or be informed no such formatter exists – and go from there. But it seems rare outside of a web context (RESTful service) that any individual usage of this would require such flexibility so it wasn’t high on my list. For now, you know what format you need and can just go with that. Anyway, I digress…
Wrapping it up
With the completion of this new parsing framework, parsing PCoIP log files going forward should now be easier to deal with. Still frustrating, but easier. At the very least, there is now a level of encapsulation and separation of duties that was missing from the organically “grown” monolithic parser that started it all. The real test will be when the log viewer breaks again, as it inevitably will, and I am required to make the first updates. That’s when I’ll know if this was worth doing. But in my view, it’s fun to make updates like this. Is my design perfect? Hardly. Are there probably a dozen libraries/frameworks I could have tapped to build this in? Almost certainly. But they often add a lot of baggage and “weight” and if you didn’t build your entire project against those frameworks it often means the benefits are limited or that you end up starting over. Building this from scratch was satisfying on multiple levels. It’s something I had wanted to do for a while, so I have the satisfaction of completing it. It’s only as large and complex as it needs to be – which isn’t very much of either. And lastly, it should allow me to support the users of this app better than I have been able to do in the past.
In the end, that’s all I really wanted.
Those of you who attended VMware Partner Exchange 2013 and came to my session on Dynamic PCoIP will be somewhat versed in this topic, and if you’ve stopped by the excellent myvirtualcloud.net and read Andre’s post on View 5.2 you may have seen my comment there and will also be up to speed.
But for those who have managed to miss out on both of those things, I wanted to recap some items here.
“PCoIP GPO settings take effect immediately when changed (host side only)”
This was announced with the launch of View 5.2 as a new PCoIP feature. But there are a few problems with this, and a lot of confusion around it (even internally to VMware).
To briefly recap the information I left in the comment at myvirtualcloud.net-
This feature allows a limited number of PCoIP settings to be applied to the registry (via whatever method) and to take effect immediately within a currently running PCoIP session. So, specifically for something like tuning, you would be able to change parameters and it would not require a disconnect/reconnect sequence for those settings to take effect. This is pretty awesome!
So what’s wrong?
Well, the press releases just generically say “PCoIP GPO settings take effect immediately” – but do nothing to specify WHICH settings. Further, some issues were introduced between the early 5.2 beta and the RTM code for View 5.2 that caused this feature to only “work” for a really small subset of parameters. In my opinion, this basically renders a potentially awesome feature mostly useless.
So, in a perfect world, what options should take effect immediately? As far as I know, the following were planned for in the original drop:
- Disable Build-to-Lossless
- Max session bandwidth
- Session Floor
- Max initial image quality
- Minimum image quality
- Max FPS
- Audio bandwidth limit
- Logging level
What options actually work in the 5.2 GA release (based upon my experience)?
- Max FPS
- Audio bandwidth limit
Slightly less useful right? So the feature is in there, but it’s not really implemented across enough options to make it terribly viable.
Why does this matter?
Having this feature added to PCoIP has been a long-standing request of mine. I’ve been building tools (sporadically, I admit) for PCoIP for the past 3 years now, and I’ve always maintained a “vision” of being able to control the protocol in real-time for both users and admins and then building new tools around that.
One tool I had started work on, but have put on hold is a real-time tuner:
This tool allows you to view content and tune at the same time; to see the subjective impact of tuning at the same time as the objective impact on bandwidth, loss, and latency. I’ve been wanting to write this tool for 2+ years! And I did, mostly, but you know what? It’s of absolutely no use to anyone right now, so I didn’t finish, and I won’t be releasing it yet.
I also have several ideas on how to improve the PCoIP Configuration Utility. With this feature fully implemented, I can create a version of it that not only stores and applies profiles, but that could do so in response to contextual data from the client. Connecting via a mobile device? Automatically apply this profile. Connecting across the WAN? Automatically apply the WAN profile. This could be keyed off IP, MAC, User, Domain – whatever! If it’s a piece of information that can be read on the desktop, it can be used as a condition to apply dynamic tuning.
That would be cool wouldn’t it?
That’s a wrap
So, to wrap this up, I’m pushing hard internally to get the initially planned set of dynamic parameters fully implemented. I am hoping to see it in the next release of View, but I can’t say if that will happen with any certainty, probably, it won’t. Still, membership has it’s privileges, so I have access to “working” code for these changes and I wanted to share a video I had intended to show at PEX 2013 but which a last minute computer failure forced me to abort. I hope it gives you a taste of what’s to come once these features are fully implemented.
I have embedded the video here for ease of access, but since youtube is only aware of video in 16:9 format you can’t view the content at 1:1 which means it’s far harder to see the visual artifacts when aggressively tuning the session. Because of that I am also going to include a download link for the source MP4 video to allow for true 1:1 playback and better subjective visual inspection.
Also, there’s a slight jerkiness to the video at times, I want to be clear that this is an artifact caused by the software-based capture method I am using and is NOT present when interacting with the View session natively in the View Client. As soon as a DVI frame-grabber capable of high-resolution capture at 30+ FPS goes on sale for less than a few grand, I will get one and this artifacting will go away, until then – sorry :-/
Direct download: Dynamic PCoIP Tuning Video
Thanks, and sorry for the lack of content recently, I am hoping to get some blog and software updates out the door in the coming weeks.