tag:blogger.com,1999:blog-61117257231347628772024-03-05T02:24:49.052-08:00XMLSHA Command line shell for XML processingAnonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.comBlogger65125tag:blogger.com,1999:blog-6111725723134762877.post-16875604774070681402013-03-25T05:04:00.002-07:002013-03-25T05:04:43.625-07:00Update to xmlsh 1.2.3<h2>
Update to xmlsh </h2>
<div>
<br /></div>
<div>
I recently pushed an update to xmlsh (v 1.2.3) and all dependant extension modules.</div>
<div>
Full release notes are here: (for the core)</div>
<div>
<a href="http://www.xmlsh.org/ReleaseNotes123">http://www.xmlsh.org/ReleaseNotes123</a></div>
<div>
<br /></div>
<div>
Just some random notes on improvements</div>
<div>
<ul>
<li>New RANDOM32 and RANDOM64 variables</li>
<li>Improved http command by switching to Apache httpclient</li>
<li>MUCH better JSON support (to be documented)</li>
<li>A log window for xmlshui makes using it as a debugger much easier</li>
<li>Improvemetns to xurl and urlencode so that you can create safe URL's much easier by passing in all the components of a URL and query as separate strings. - Critical for creating complex REST calls.</li>
<li>various bug fixes</li>
</ul>
<div>
<br /></div>
</div>
<div>
In addition to core, some enhancements to aws, marklogic, and twitter extensions.</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-4390017609163697372013-01-22T05:20:00.000-08:002013-01-22T05:20:07.264-08:00JSON SupportToday I added the beginnings of native JSON support in xmlsh. This is in SVN only, no new releases yet. I am soliciting oppinions on features for JSON.<br />
<br />
Why JSON in xmlsh ? Well I have to learn to stop hating and "Love the bomb".<br />
JSON is prevalent in the web world. I want to add better support for web services. While the existing json2xml command works fine it is a bit clumbsy. So I decided to support native JSON parsing with "jsonread" and JSONPATH with the "jsonpath" command. Also variables can be pure parsed JSON objects. <br />
<br />
From here what ? This is a good start but I would like the equivilent of <[ xquery ]> but to work on JSON. Maybe thats overkill ? How about simple array and member access natively like $X.Y[2]<br />
<br />
Not sure how far I want to go with this.<br />
<br />
Suggestions welcome !<br />
<br />
-DavidAnonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-7827005090463783632012-08-13T10:20:00.001-07:002012-08-13T10:20:41.674-07:00Change is coming to the Markup WorldA small post about my experiences after attending Balisage 2012.<br />
<br />
<a href="http://blog.calldei.com/2012/08/big-changes-in-th.html">http://blog.calldei.com/2012/08/big-changes-in-th.html</a>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-11987284536784065022012-07-26T09:00:00.001-07:002012-07-26T09:01:05.184-07:00Balisage 2012<h2>
Balisage 2012 is quickly approaching !</h2>
<div>
The annual congregation of markup geeks is quickly approaching. </div>
<div>
<a href="http://www.balisage.net/">http://www.balisage.net/</a></div>
<div>
<br /></div>
<div>
I will be attending as well as presenting Wed August 8 at 2:00 pm (<a href="http://www.balisage.net/2012/Program.html">http://www.balisage.net/2012/Program.html</a>). The schedule looks great.</div>
<div>
<br /></div>
<div>
In addition there will be a MarkLogic <a href="http://www.balisage.net/2012/DemoJam.html">DemoJam</a> on Tuesday Aug 7, free booze and food and a chance to win great prizes.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
This is the Must Go To conference of the year. Hands Down.</div>
<div>
What are you waiting for ! Register now.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-43958641254147748232012-06-14T11:41:00.000-07:002012-06-14T11:41:20.681-07:00Installer for xmlshInstallers are pretty cool things. They just do what you want and don't bother you with knowing how. However they are also "black boxes" and tend to be dependent on things like having a GUI or a particular OS and you never know what they actually do under the hood.<br />
<br />
For a product like xmlsh I never thought much about an installer. Its so eazy, just unzip the distribution, set a few env variables, maybe chmod a file , oh and dont forget the pixie dust. By not having an installer I can ignore all that and give you the warm fuzzy feeling that I'm doing any magic dirty tricks behind your back. Its just files right ?<br />
<br />
But then its not always obvious. An installer is, well, convenient! So I've been looking into java installers and found a few. But before spending any time on it I was wondering if my wonderful captive audience could make any suggestions. Would an installer actually help anyone ? Were you daunted by the obscure installation instructions ? Do you have a GUI always available (X windows or Mac or MS Windows) ?. Does an installer feel more friendly or more opaque ?<br />
<br />
Inquiring minds want to know !Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com3tag:blogger.com,1999:blog-6111725723134762877.post-72197683834539887282012-06-14T11:40:00.003-07:002012-06-14T11:40:58.122-07:00Release 1.2.0At long last many things came together and I have released xmlsh 1.2.0 along with updates for all the extension modules. This is a really big release and I havent had time to document it all ...<br />
But just to get you excited<br />
<br />
xmlshui - a simple GUI for xmlsh<br />
Named Pipes - xmkpipe creates named pipes for either text or XDM Streaming<br />
XDM Streaming - using named pipes or implicit pipes (| with the set -xpipe)<br />
Streaming enhancements - Some of the core commands now can stream unseralized XDM through named or implicit pipes - xsplit, xsql , marklogic:put ... more on the way. <br />
Lots of bug fixes and test case improvements.<br />
Marklogic extension improvements - XDM Streaming, dynamic URI construction using {seq} or {random}<br />
<br />
This is a solid new release with extremely powerful new features and improvements on the core features. I will be spending the next few weeks updating the docs to help explain how to make the best use out of the new features.Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-49254908194772106992012-05-31T17:07:00.001-07:002012-05-31T17:07:38.891-07:00GUI GUI who wants a GUI ?At long last (years) I knuckled down and made a simple xmlsh GUI. This will be in the next release. <br />
Why ? I have been opposing this for a long time for many reasons, the least of which is I dont really like writing GUI's. But I got tired of the limited editing capabilities of DOS command shells and enjoy the very simple BSH UI GUI ...<br />
I think what stopped me for long is the slippery slope. Once you start with a GUI where do you stop ? Xmlsh is a command line shell and a embedded API, not a WYSIWYG tool.<br />
But alas ... a simple GUI is useful sometimes. I played around with various toolkits and settled on using plain AWT and Swing. I found that Eclipse Windows Builder supports simple AWT apps. Quite a nice tool. I tried SWT but while it is much more feature full, it is very much tied to Eclipse and required a dozen more jar files to run even a basic window. With AWT I was able to do a functional GUI in 200 lines of code. This will likely expand to 20000 ... as feature creep sets in, but its a start.<br />
<br />
Here is a screenshot of a sample session "xmlshui"<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeaB141M_i1TW8DhycNsCnb-g4ENL3auieGB77MPs-ebgr2wi3d96JQ1pDNSTg3tYkuQk264Y4gDiy7WjfbYxnlp5mIQCoNz2T-7LgBDvTPhyphenhyphenqy7wDcPihid3LKgIHqHH3Kn2V4QqgECM/s1600/xmlshui.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="547" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeaB141M_i1TW8DhycNsCnb-g4ENL3auieGB77MPs-ebgr2wi3d96JQ1pDNSTg3tYkuQk264Y4gDiy7WjfbYxnlp5mIQCoNz2T-7LgBDvTPhyphenhyphenqy7wDcPihid3LKgIHqHH3Kn2V4QqgECM/s640/xmlshui.PNG" width="640" /></a></div>
<br />Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com1tag:blogger.com,1999:blog-6111725723134762877.post-53301399573788751602012-05-16T05:24:00.000-07:002012-05-16T05:24:59.034-07:00Streaming Streaming Streaming ...Now that I work for <a href="http://www.marklogic.com/">MarkLogic</a> I am dealing with more and more "Big" "Data" ... and as usual xmlsh + marklogic is a huge win. But as I start ramping up my use of large datasets especially large numbers of small documents (millions, hundred million ...) the old tricks dont work quite so well.<br />
<br />
For example recently I needed to upload 3 million XML files to a ML server from a relational DB.<br />
My first pass was my favorite tool for this ... xsql + xsplit + ml:put<br />
Since I like to debug stuff as I build it ... the simple way is to do this.<br />
<br />
xsql ... > bigfile.xml<br />
xsplit -o xml bigfile.xml<br />
cd xml<br />
ml:put -baseuri /xxx/ -m 100 - -maxthreads 4 *.xml<br />
<br />
On my big beefy server box this worked although a bit slowly. So ok I wanted to now transfer this data to an EC2 instance. Its "only" 10G of data so I did this<br />
<br />
tar -cvzf xml.tar.gz xml<br />
<br />
then transfered the now compressed file to the EC2 machine.<br />
Then on the EC2 machine I tried to replicate the above steps.<br />
<br />
tar -xzf xml.tar.gz <br />
<br />
I waited ... waited ... waited ... 3 DAYS and it wasnt done yet. Admitedly this was a medium instance of EC2 but it should have handled this. The problem seemed to be the system was stuck in 90% system time.<br />
My guess is the age old problem of lots of files in a directory. Especially over EBS ... it just doesnt perform well. Its actually exponentially slow to add files to directory once they get big ... particurly nasty when the files are small so the overhead of simply creating a file entry is much bigger then the file IO itself.<br />
<br />
So what to do ... I did 2 things ... I restarted the EC2 instance as an m1.xlarge ... ($$$$ ka chink)<br />
Then instead of pre extracting the xml to a directory in whole I used a new feature I recently added to ml:put ...<br />
<br />
tar -xzf xml.tar.gz | ml:put -baseuri /xxx/ -m 100 -maxthreads 4 -f - -delete<br />
<br />
What this does is let tar still extract the files but it then lists them to stdout.<br />
From there ml:put reads the list of files as they are extracting, batches them up and sends them to MarkLogic then deletes them. The end result is that there is only about 500 or so files in the xml directory at any one time. This completed in about half an hour ... about 2000 docs/sec ... much better.<br />
Of course this speedup was due to the larger instance as well as the technique ...<br />
<br />
But this gets me thinking ... Why do I need the overhead of writing to a temp directory for this ? Its still adding a significant unnecessary overhead. I should be able to send a bunch of XML files to ml:put in a stream and use no temporary files. In fact I should be able to do a full pipeline with no overhead like<br />
<br />
xquery 'for 1 to 10000000 return document { ... } ' | ml:put ...<br />
<br />
or perhaps<br />
<br />
xsql 'select * from table' | xsplit -stream | ml:put ....<br />
<br />
The core problem here is the lack of a streaming interface for XDM. In order to send a bunch of XML files (or XDM values) through a stream (or to a file and back) they need to be packaged in something. Typically wrapped in a root element or maybe zipped or tar'd.<br />
<br />
Zip is really lousy for this because its TOC is at the end so you cant stream unpack a zip file. Tar is good because each file entry is contigous and you can stream unpack them. But what about cases where I just want to dynamically create (or transform) XML and spit it out like the first example<br />
xquery 'for 1 to 10000000 return document { ... } ' | ml:put ...<br />
<br />
<br />If I wrap this in a single document it becomes hard to stream. ml:put *could* have xsplit builtin ... but to keep to the tools approach I'd rather split the functionality. So say I put xsplit into the pipeline like the second example. How is xsplit to produce *multiple* documents on its output stream in a way that is readable ? Were back to a serialization format for XDM (<a href="http://xml.calldei.com/XDMSerialize">http://xml.calldei.com/XDMSerialize</a>)<br />
<br />
This is a fundimental problem in traditional XML toolchains. There is simply no standard and efficient way to stream sequences. So what to do ?<br />
<br />
I'm considering a 3 phase approach.<br />
1) Implement an enhancement to xmlsh commands and pipes such that they can request, produce, and consume sequences through ports. So for example "xsplit -stream" could output the split documents all to stdout. But what would this look like ? How to implement it ?<br />
<br />
2) For pipes implement an optional XDM stream pipe. This would allow streaming of XDM values (including sequences of documents), without serialization directly through the pipe. This does mean that the pipe might get large if the documents are large ... I may have to limit the pipe to a small number of values.<br />
<br />
3) Implement some kind of text serialization for sequences. Essentially back to <a href="http://xml.calldei.com/XDMSerialize">http://xml.calldei.com/XDMSerialize</a> ... although I am not sure I like my proposal so much in the face of this use case. The original proposal does not consider streaming as the major use case. However the use cases it was designed for should overlap with streaming. I'm not even sure I need to support most of XDM ... falling back to what XProc does (streams of documents) may be sufficient although I abhor the restriction on purely theoretical grounds. But the fact is any text serialization of XDM will be lossy. It is just a matter of drawing the line somewhere, and maybe the most valuable use case is drawing the line at documents.<br />
<br />
<br />
Well back to the drawing board. I'd like to implement this but still so many open issues !!!<br />
Comments welcome.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com5tag:blogger.com,1999:blog-6111725723134762877.post-45732311354091279082012-04-09T11:46:00.001-07:002012-04-09T11:46:52.279-07:00Spring Update<h2>
Spring Update</h2>
<div>
I have updated xmlsh to verson 1.1.9. The main enhancement is upgrade to Saxon 9.4.</div>
<div>
Also updated the MarkLogic extension.</div>
<div>
</div>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-86753446700116277332011-12-15T14:06:00.000-08:002011-12-15T14:06:04.338-08:00XMLSH v 1.1.8<b>Winter Update</b>
I've updated xmlsh and all the extension modules (both documented and non documented) including
<br />
<ul>
<li>MarkLogic</li>
<li>eXist</li>
<li>JSON/JXON</li>
<li>AWS</li>
<li>JMX</li>
<li>Calabash/XProc</li>
</ul>
<div>
<br /></div>
<div>
Primarily a minor tweek and bug update but also includes new commands and features and updates to the latest run-times of all extension modules (MarkLogic 5.0 , latest Calabash , latest AWS etc).</div>
<div>
<br /></div>
<div>
Definitely a suggested upgrade for all.</div>
<div>
<br /></div>
<div>
One caveat. Due to a suggestion from a reader, as well as my long-term wish, I've changed "xls" to use a different tag for files and directories. More consistent, didn't break any unit tests but might break your scripts. Sorry ... </div>
<div>
<br /></div>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-80226503415426801092011-09-28T06:27:00.000-07:002011-09-28T06:31:54.709-07:00GUI for xmlshI've long considered implementing a GUI for xmlsh.<br />I didnt do so from the start because I didnt want xmlsh to *be* the GUI (or require one). But now that the core is stable and mature, there are times when a GUI would be very useful.<br /><br />Suggestions (and help!) welcome for this upcoming project.<br /><br />Ideas I have<br /><br />* optional. Core xmlsh not affected adversly<br />* portable. Probably (java) means using AWT or Swing ? <br />* atleast 1 mode that simulates a typical terminal with line editing<br /><br />Onto things which might be really neat<br />* GUI view of variables <br />* debugger<br />* syntax sensitive source browser/editor<br />* multiple windows (one per thread ?)<br />* Eclipse plugin ?<br /><br />A lot of things (and wasted time) could be put into this. I'd love feedback as if any of this seems useful to you.Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com4tag:blogger.com,1999:blog-6111725723134762877.post-50625330585250680962011-06-03T08:16:00.002-07:002011-06-03T08:18:17.804-07:00Released developer edition of eXist extensionI have released the 0.1 "developer" edition of the eXist extension module for xmlsh. This is Pre-Alpha quality and should not be used in production. <br /><br />http://www.xmlsh.org/ModuleExist<br /><br /><br />Comments welcome ! There's a lot left to go with this, but it supports the core REST operations exposed as DB operations put/get/invoke/query/del and one example list. From these I should be able to build a comprehensive set of tools for eXist.Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-48866548117920778082011-06-02T08:56:00.000-07:002011-06-02T08:58:02.867-07:00Released update to MarkLogic extensionThanks to some help from the field I found and fixed a bug in the MarkLogic invoke command. When using the "-v" option to pass external variables to stored xquery's the arguments were being misread.<br /><br />This has been updated as version 1.12 of the MarkLogic extension module ( 2011-05-02)Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-90191316328187163152011-05-05T18:00:00.000-07:002011-05-05T18:11:27.857-07:00Release 1.1.5I've been really lazy and haven't posted on this blog for a while. Even skipped the last release. <br /><br />Today I released xmlsh 1.1.5 as well as updates to the MarkLogic and Calabash extension modules. The main feature is updating to Saxon 9.3. Also includes some bug fixes, fixed demo app, improved test cases.<br /><br />So whats coming up next ? I have a LOT of things in the pipeline. A major extension module I hope to release this year is the JSON extension module. This is an implementation of the JXON processor and associated tools. You may notice the json2xml and xml2json commands have been updated to use the <a href="http://xml.calldei.com/JsonXML">JXML</a> schema. This is just a tiny part of the JXON processor. I hope to release this in time for the upcoming <a href="http://www.balisage.org">Balisage 2011</a> conference. But if your curious now, an early implementation is checked into sourceforge. <br /><br />Also in the pipes are extension modules for the <a href="http://exist.sourceforge.net/">Exist XML DB</a> and also for <a href="http://aws.amazon.com/">Amazon Web Services (AWS)</a><br /><br />No ETA on these yet but I've started work. Volunteers are welcome !<br /><br />I'm also experimenting with the PE and EE Editions of Saxon. Some really great stuff in there especially XQuery 3.0, XPath 3.0 and XSLT 3.0. Unfortunately these are all paid/licensd features so I am reluctant to require their use. However I have tested xmlsh with Saxon 9.3 in both PE and EE versions to make sure you can use these features. I'd really love to have them as part of the core technology but not yet willing to pull the rug on a full free open source implementation.Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com1tag:blogger.com,1999:blog-6111725723134762877.post-18190940686945427622011-01-17T04:28:00.000-08:002011-01-17T04:30:11.242-08:00XMLSH 1.1.3 released, 1.1.4 coming soonThanks to some great feedback from users as well as my JSON/XML project I've been making significant improvements, mostly fixes, to xmlsh.<br />1.1.1 and 1.1.2 , 1.1.3 were released in Dec, Jan. And a 1.1.4 is coming soon.Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-35353835359453439132010-11-18T08:35:00.000-08:002010-11-18T08:37:08.374-08:00xmlsh 1.1 releasedI've finally gotten xmlsh back into good robust state after adding a bunch of new features.<div>Version 1.1 is here !</div><div>I have not yet completed documenting all the new features, working on that over the next few days, but you can look in the test cases for examples.</div><div><p class="MsoPlainText"><o:p></o:p></p> <p class="MsoPlainText"><o:p> </o:p>Major new features include</p><p class="MsoPlainText"><o:p></o:p></p> <p class="MsoPlainText"><o:p> </o:p>* Scriptable Streaming XML with StAX functions (at the script level)</p><p class="MsoPlainText"><o:p></o:p></p> <p class="MsoPlainText">* Native functions <o:p></o:p></p> <p class="MsoPlainText">* Function call expression syntax. eg. echo foo( bar ) <o:p></o:p></p> <p class="MsoPlainText">* Native Java object creation, variables, and method calls<o:p></o:p></p> <p class="MsoPlainText">* Significant performance optimizations including reworking many commands from non-streamable to streamable<o:p></o:p></p> <p class="MsoPlainText">* JSON / XML conversions (preview feature, still in progress)<o:p></o:p></p> <p class="MsoPlainText"><o:p> </o:p></p> <p class="MsoPlainText"><o:p> </o:p></p></div>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com1tag:blogger.com,1999:blog-6111725723134762877.post-71052838716580192642010-10-12T10:31:00.000-07:002010-10-12T10:34:03.383-07:00Function syntax for native java objectsI'm making some good progress on integrating java objects.<div>I now have function syntax working for object variables.</div><div><br /></div><div>Here's an interesting example of mixing functions, jset, and the function syntax</div><div><br /></div><div><div># Define a function jnew which creates a String </div><div>$ jnew () {</div><div>local _x ;</div><div>jset -v _x -c java.lang.String "$@"</div><div>return $_x;</div><div>}</div><div><br /></div><div># Create a string object</div><div>$ a=jnew("hi there")</div><div><br /></div><div># Check its value</div><div>$ echo $a</div><div>hi there</div><div><br /></div><div><br /></div><div># Check its type</div><div>$ xtype $a</div><div>java.lang.String</div><div><br /></div><div><br /></div><div># Call the length method</div><div>$ echo a.length()</div><div>8</div><div><br /></div><div># Call the concat method</div><div>$ echo a.concat(" some more")</div><div>hi there some more</div><div><br /></div></div><div><br /></div>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-402604427051719372010-10-07T11:04:00.000-07:002010-10-07T11:06:40.009-07:00import of jar files / setting classpathIn combination to the support of native Java objects, the import command will now have a new option, "java". This appends to the classpath of the current shell and makes the jar file or directory available to all commands which use the classloader (such as xsql, jcall , jset , modules )<div><br /></div><div>Example:</div><div><br /></div><div>import java myfile.jar</div><div>jset -c mypackage.MyClass -v var</div><div><br /></div><div>Will look in myfile.jar (in addition to the global classpath) for mypackage.MyClass</div><div><br /></div><div><br /></div><div><br /></div><div><br /></div>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-81809657098513245022010-10-07T10:55:00.000-07:002010-10-07T11:03:51.596-07:00Native Java supportA new project has let me to finally realize something I wanted to do for some time. That is direct support for native java objects as expression values.<div><br /></div><div>Currently there is the "jcall" command which supports calling the main method of a java program in the same JVM. This is useful to avoid a new java process overhead, but is not that useful to get at native java objects and methods which are not already exposed to the language.</div><div><br /></div><div>There is also extension modules which allow you to write custom java code and expose it as commands. However the types of values going into and out of the modules are still limited to the native types of xmlsh (XDM values).</div><div><br /></div><div>There is use for being able to pass in native java Objects to some commands. For example the xsql command currently requires connecting to the database every call, but if you could create, persist, and pass in a Connection object it could be reused for multiple calls.</div><div><br /></div><div>Now you can. Not yet released, but checked into the source repository is support for allowing any java Object to be passed as an expression or stored in a variable.</div><div><br /></div><div>But how do you *get* these java objects created and assigned ?</div><div>One way is to pass them in through the calling application, setting them as an XValue to the shell positional parameters or environment variables.</div><div><br /></div><div>Another way is a new command "jset" which lets you create java objects, set them to variables and then call methods on the objects and assign those to other variables.</div><div><br />Example:</div><div><br /></div><div>jset -v str -c java.lang.String "Hi There"</div><div><br /></div><div>will create a java String object and assign it to $str</div><div><br /></div><div>You can do the same for Date, for example</div><div><br /></div><div>jset -v date -c java.util.Date</div><div>echo $date</div><div><br /></div><div>You can also call methods (static or instance).</div><div><br /></div><div>jset -v len -o $str -m length</div><div><br /></div><div>sets $len to the result of String.length() </div><div><br /></div><div>I'm still working on a better syntax, possibly with combination to the "tie" command so you can tie method invocation to variable expansion.</div><div><br /></div><div>I'd like to see something like ${str:length} call the length() method of the object in $str</div><div>but I need to flesh out issues like passing in arguments.</div><div><br /></div><div>Be ready for a new release soon with this support ! Suggestions as always are welcome.</div><div><br /></div><div><br /></div>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-28375146205410243752010-06-25T08:56:00.000-07:002010-06-25T08:59:30.393-07:00Functions are comingI've implemented an experimental function call expression syntax. This is coming in Version 1.0.7 (ETA by July).<br /><br />See<br />http://www.xmlsh.org/SyntaxFunction<br /><br />For some documentation on how this will work.<br /><br />Considering also adding dynamic type coercion so that variables interact with xquery expressions in a more obvious way.<br /><br />E.g.<br /><br />a=1<br />echo <[ $a + 1 ]><br /><br />Fails because $a is a xs:string type.<br />You need to do <br /><br />a=<[1]> <br />echo <[ $a + 1 ]><br /><br />to get an integer<br /><br />I'm considering having expressions passed to xquery or perhaps when evaluated to automatically be coerced to the apparent type, but I haven't worked out all the ramifications of this yet.Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com1tag:blogger.com,1999:blog-6111725723134762877.post-71998805286916439312010-05-24T18:39:00.001-07:002010-05-24T18:44:26.681-07:00Better error diagnostics comingIn the next release of xmlsh (ETA end of may), there is a significant improvement to error diagnostics. This effects normal function errors, as well as the -v and -x options. <br /><br />All command errors that cause a usage or exception, as well as -v and -x output <br />produces file/line diagnostics. <br />Example<br /><br /><br />$ echo -foo<br />[stdin line: 1]<br />echo: Unknown option: foo<br /><br />In scripts this includes the filename of the script and line number.<br /><br />This output is also in -v and -x output (preceeding the function)<br />Example:<br /><br />$ set -v -x<br />$ echo foo<br /><br />- [stdin line: 7]<br />echo foo<br />+ [stdin line: 7]<br />echo foo<br />fooAnonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-33883721647446953862010-05-15T12:07:00.000-07:002010-05-15T12:24:53.227-07:00Functions and more functionsThanks to some great discussions with Dave Pawson, I'm considering extending the syntax for function calls and the possible return values for functions, scripts, and commands.<br /><br />Right now a command, script, or function can only "return" an integer, often known as the "Exit Status". Historically this is because C programs (and most OS programs) "exit" or "return" with a single integer value. On older systems this was actually limited to a byte. 0 means success, non-zero means fail.<br /><br />So things have been for 40 years ... <br /><br />But with xmlsh its an artificial limitation. Why make functions and scripts limited to int return values ? I cant do anything about external programs, but functions and scripts *could* return any XDM type. Why not let them ?<br /><br />After much discussion I've concluded that the only thing in the way is a syntax to capture the return value which doesn't break compatibility with capturing the stdout.<br /><br />The typical syntax for capturing the "output" of a command/script/function is $(cmd).<br />This captures the stdout of the command and converts it to a string.<br />XMLSH has extended this a bit with $<(cmd) for XML types and cmd >{var} as a synonym. But both extensions still only capture the stdout.<br />The only way to get the return value is either using $? or using the function in a boolean context like<br /><br /> if command ; then<br /><br />In which case the exit status of the command is interpreted as a boolean (0=success).<br /><br />But why not let functions (and possibly scripts and internal commands) "return" non-integer values ?<br /><br />Like this<br /><br />function concat() <br />{ <br /> return "${1}${2}"<br />}<br /><br />What are the implications ?<br /><br />For one, $? could be any type not just integer. I dont foresee any big problems with that.<br />e.g.<br /> concat foo bar <br /> a=$?<br /><br />produces "foobar" in $a<br /><br />Existing code that assumed functions returned itegers would still work as long as those functions or commands were not changed.<br /><br />What about boolean context ? <br /> <br /> if concat foo bar ; then ...<br /><br />I'd need to define some compatible conversions from any XDM type to boolean.<br />These couldnt *quite* be the same as the xquery conversion to xs:boolean because I'd want 0=true and integer != 0 false.<br />But other then that most conversions could be fairly obvious<br /> () => false<br /> non-empty sequence => true<br /> xs:integer => see above (0=true)<br /> "" => false<br /> non empty string => true<br /> node/element/document => true<br /><br />should be OK.<br /><br /><br />But why do all this ? One more step is desired. The ability to call functions and make them "look like function calls" ; instead of relying on $?<br /><br />why not support something like<br /> concat( foo , bar )<br /><br />This could then nest <br /> concat( concat( foo , bar ) , spam )<br /><br />This syntax func(...) would evaluate to the return value of the function. The stdout would still go to the parents stdout so you could pipe through them<br /> <br /> func(a) | func2(b) <br /><br />Then there are some details about globbing, and argument lists. If I support the "," as an argument seperator (instead of space for command invoction) then this brings some interesting side effects<br /><br /> foo( *.xml ) => $1 becomes a sequence of xs:string<br /> foo( *.xml , *.c ) => $1 , $2 both sequences<br /><br />and so on ... this could have some really interesting side effects. <br />Consider if commands as well as functions could be called this way. Why not ?<br /><br /> cat( foo.xml , bar.txt ) <br /><br />as syntatically equivilent to<br /><br /> cat foo.xml bar.xml<br /><br />This would allow more "programming like" language style to be used. It also opens the doors for function signatures (or command signatures) for static or dynamic type checking and variable assignment.<br /><br />For example, suppose you'd write a function now like <br /><br />function copy() {<br /> from=$1<br /> to=$2<br /> cp $from $to<br />}<br /><br /><br />Adding a signature could be closer to xquery or java or C<br /><br />function copy( from , to )<br />{ <br /> cp $from $to<br />} <br /><br />Then callers of copy could produce an error automatically if they provided the wrong number of args<br /><br /> copy a b c # -> runtime error by interpreter<br /><br />could add to that types eventually <br /><br />function copy( from as xs:string , to as xs:string )<br />{ ... }<br /><br /><br />then the next obvious step is return types as part of the signature<br /><br /><br /><br />Anyway something to think about ... <br />I suspect I'm going to try the first step and see what breaks. Let return take any type , and let $? take on any type. <br /><br />Lets see if that breaks anything first ... <br /><br />Comments welcome !Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com2tag:blogger.com,1999:blog-6111725723134762877.post-4533896822613236832010-05-06T13:31:00.001-07:002010-05-06T13:32:09.414-07:00Annonnce: xmlsh and marklogic extension update releaseI have updated xmlsh (version 1.0.4) and MarkLogic extension to xmslsh (version 1.1) on sourceforge.<br /><br />https://sourceforge.net/projects/xmlsh/<br /><br />I actually included a changelog.txt for the first time !Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com0tag:blogger.com,1999:blog-6111725723134762877.post-48947080763163229742010-04-20T08:57:00.001-07:002010-04-20T08:58:55.599-07:00Updates coming, new commands, better servlet code, better MarkLoigcJust a quick note to say new commands are coming.<div><br /></div><div>* base64 (posix)</div><div>* xmd5sum (md5 sums of files)</div><div>* xunzip (unzip and list zip archive in xml friendly way)</div><div><br /></div><div><br /></div><div>Improved servlet integration.</div><div>Much improved marklogic integration, </div><div>* multi-threaded put</div><div>* commands for properties and permission control</div><div><br /></div>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com2tag:blogger.com,1999:blog-6111725723134762877.post-2966721375028701602010-03-21T15:40:00.000-07:002010-03-21T15:43:21.945-07:00xmlsh 1.0.3Released xmlsh 1.0.3 today.<div><br /></div><div>This includes</div><div><br /></div><div><ul><li>enhancements to xgetopts</li><li>enhancements to XML Servlet (headers and parameters)</li><li>new command httpsession to get/set session parameters</li><li>Many bug fixes</li><li>{expr} syntax in command line arguments to preserve sequences</li></ul><div><br /></div></div>Anonymoushttp://www.blogger.com/profile/06155161084044858311noreply@blogger.com1