Which Scripting Technology?

This has been my biggest headache and the tempation to just pull in the old VBScript modules have been very tempting. The main reason for this is that it is the simplest form of scripting that I can see without going into the world of 3D/2D visual representations – and I am really tempted I can tell you. A lot of the old ideas have been haunting me and if I had more time I definitely would have dived in – the idea of having a scripting process so simple to use that a child could do it has always had an appeal to me and I certainly would love to spend the time  seeing what others may have done in this area.

Well the alternatives seem to favour going for JavaScript, mainly because of the strong and tremendous work that has gone in behind GreaseMonkey and Userscripts.org. Add to that that JavaScript tends to be favoured by Google Scripts too. The down side of course maintaining simplicity.

For a while I thought maybe PHP may be a way to go allowing the crossover from scripts run locally and on the server.

The courts still out and I am jumping back occasionally to research it further. If it takes too long I might as well include all of them, but the down side will be the confusion it might cause and maintaining the application. I need some expert opinions in this I think… The worst thing for time and effort is to change it during development, but it may need to happen because really I want this application back up and working for me as soon as possible.

Absolute Uniqueness Algorithms

This has been an area where I have deliberately spent extra time. I figured that the output from the spinning functions need to be as unique as possible including the ability to create, recreate, stop and start the process anywhere. So I have come up with a few algorithms for maximising uniqueness. The only problem I have at the moment is deciding on how to measure it – there are so many applications out there that widely differ in this and I want the best definition of it that I can find.

Also I have pushed to be as modular as possible with any functions here. To be able to spin text or any document possible to files, zips or clipboard (almost mentioned DDE – that’s a sign of age) is part of the idea and I must leave the hooks there to be able to plugin any inputs and outputs that I can think of. I want to leave it completely open and available for anyone to add modules whenever needed. I have even looked at some of the video creation ideas that exist out there – so keeping each function in its own domain is a must.

Long term too I want to create assembly for the main routines, although at the moment I have succumbed to the ease of .NET handling I still have proper flowcharts in my mind as to what is going on and its got to be as fast as possible using the least resources. At the moment resources get used with no regard so the first versions of this will just focus on function not efficiency.

The Eventual Purpose of QKSite

Need to remind myself that the spinning that exists there has to migrate into a plugin of QKSite. It is not of itself meant to be a article spinning application, this was done just so I could use it soon myself. The spinning functionality will disappear into a module or plugin of some kind, depending on the final architecture and be brought into any scripts or language only when it is required. The main purpose of QKSite is to manage it and allow automation to be created, shared, controlled, scheduled and visibly manipulated as kinds of projects. For me, for now I see it visually ending up as a kind of dashboard of all the things going on, with the most effective summary views of any activity being shown immediately and constantly. The ability to see what events are happening and to react to them immediately is important, then being able to either deal with them from the dashboard or drill down into the information, to manipulate or query it at any level right down to source.

In a specific visual idea of this, I want to see what is happening on all my sites, all my social and communication points at any time and just control it all from one place. Yes, sounds a bit like a horrendous task, but I am in no hurry at the moment. Other things like keeping the wolf from the door are enought to occupy me for the time being…

So let’s think of an agreed signature for a document. The rule should be a template and a signature should be all that is needed to create the final document. So the simplest signature possible (for spinning only at the moment) should be a recording of which snippets were used to create the output. Snippets are the strings between the delimiters, the places in the document that would actually change.

So in the following example, if the template is

A {Day|Week} In {A|The} Life

The only possible outputs would be

A Week In A Life
A Day In A Life
A Week In The Life
A Day In The Life

So the simplest signature would be which options were chosen in both snippets that were spun. The signatures for the above lines would be

2,1
1,1
2,2
1,2

respectively.

This is easily read and understood and allows a reader to work out what was used where so it looks like a good idea to stick to it and make it exactly that format for now. Maybe later for efficiency the bit arrays can be fully used but for now, to get this working, we’ll store them in that fashion.

Although the situation is more complex when you want a spin within a spin I found parsing a template into an array of snippets to be really helpful and dealt with it easily. Each time the delimiters are found they are add to an array of snippets. If the snippets themselves contain delimiters then more snippets will be created to deal with them.

So the following template will produce an array of three snippets

This product is {top|the {highest sales|best performing} widget} of its range

0: This product is {1} of its range
1: top|the {2} widget
2: highest sales|best performing

So the signature mask of this template will contain 2 numbers, for snippets 1 and 2. The first snippet did not have any split ‘|’ delimiter so needn’t be recorded. Therefore the total number of signatures for this template is exactly the same as the earlier one, with only 4  possibilities.

What are the advantages of these signatures?

Well their neat fit with the template means that you could actually stored efficiently the documents that have been produced. More efficiently if bit array are used but that’s for later. So you would have a template and a list of signatures that define what documents have already been done, meaning you can continue in separate stages. You could for example generate a new unique document some time later, whenever you needed it, say for example submitting to directories over an extended period of time.

It means also that you do not need to produce everything at once and only create these documents when they are needed. Useful if you are using several different documents in one go to, without having to produce all the output in marathon sessions.

Another advantage comes back to ensuring uniqueness in the application, the signatures themselves can be the key to ensure that the application does not reproduce the same article again by just checking the existing signatures. I still need to work this out in a future post, but I have had a few ideas for that to keep it efficiently simple, as best I can.

The biggest advantage is that the signatures use less memory, they are smaller, and could possibly be kept in a spreadsheet or database to show which ones have been produced.

Another advantage is that the same output document can be produced again when needed. By applying the signature to the template, meaning that practically creating loads of output files is not really necessary and you can look back at earlier documents that you produced.

What are the disadvantages?

Well, I have been looking around to find the best technology for scripting and to put them into the article template. This is actually the most important feature personally that I am looking for to using, because although spinning give colour and uniqueness to the final document, if used correctly, the most power comes when the document is powered by the data you have collected.

So also it means that uniqueness in signature is of less importance, because the material then gets fed from data that is constantly changing. Also it will break the rule of  “a template and a signature should be all that is needed to create the final document” I stated in the beginning.

The thing that would need to be done is to add in the (smallest subset, decision point of, or smallest set of the decision funnel) data that would be needed to recreate the output article. There are more interesting ideas there but we’ll leave that to the back of my mind for the moment.

Actually the most troublesome bit at the moment is trying to decide on the technologies to use… Things are not as clear cut as they were a few years ago and I get this urge to bring back a 10 year technology VBScript, just because of its simplicity for any users…

Yet another post (or more) there, I am sure.

So Version 1.0.2 will have signatures, they will be a useful feature going forward.

You can find it here – http:\\downloads.qksite.com\setup.exe

Here is the first pass parsing code

    Private Sub ParseIt(ByRef strTemplate As String)
        Dim icur As Integer = 0
        ReDim arrStrings(0)
        ReDim arrBelong(0)
        arrStrings(0) = ""
        arrBelong(0) = 0

        ' First pass scan
        For x = 0 To Len(strTemplate) - 1
            Dim strChar As String = strTemplate.Chars(x)
            If strChar = "{" Then
                Dim iprev As Integer = icur
                icur = arrStrings.Length
                ReDim Preserve arrBelong(icur - 1)
                arrBelong(icur - 1) = iprev
                ReDim Preserve arrStrings(icur)
                arrStrings(iprev) += "{" + Str(icur) + "}"
                arrStrings(icur) = ""
            ElseIf strChar = "}" Then
                icur = arrBelong(icur - 1)
            Else
                arrStrings(icur) += strChar
            End If
        Next

        ' Clean up the white stuff
        For x = 0 To arrStrings.Length - 1
            arrStrings(x) = arrStrings(x).Trim
        Next
    End Sub

Which is done once at the beginning and pulls out the strings that need to be processed on in a template. Although a little tricky keeping track in the arrays is pretty much straightforward.

The spinning however became a little more complex and I kept it reasoably simple to test it out.

    Private Function SpinIt(ByVal strPart As String) As String
        Dim strParts() As String = strPart.Split("|")
        Dim random As New Random()
        Dim strResult As String

        If strParts.Length > 1 Then
            ' Choose which bit randomly
            strResult = strParts(random.Next(strParts.Length)).Trim
        Else
            strResult = strPart
        End If

        ' Now process descendants

        While strResult.IndexOf("{") > 0
            Dim iStart As Integer = strResult.IndexOf("{")
            Dim iEnd As Integer = strResult.IndexOf("}")
            Dim iLoc As Integer = Val(strResult.Substring(iStart + 1, iEnd - iStart - 1))
            strResult = strResult.Substring(0, iStart) + SpinIt(arrStrings(iLoc)) + strResult.Substring(iEnd + 1)
        End While

        Return strResult
    End Function

The fault with this at the moment is in the randomness, admittedly I didn’t put any effort in there, but it does need to be fixed so that two outputs will never be the same.

What do we need?

Speed – we want it to be as efficient as possible at the quickest speed possible.

Size – we want to handle templates that may be extremely large, I am thinking ebooks or other media later.

Flexibility – we want to be able to ensure that all inter-related functions within a template are dealt with. This means they should always all get resolved fully but only when they are likely to be needed. If a function makes a call to a data warehouse or database then it should only happen if that part of the template will actually get spun. Otherwise you will get unnecessary calls.

Simple passing through the template and resolving each of the delimiters is the very simplest method, but with spins within spins we need either multiple passes or recurse. My first impression was to go the easiest route and even though it would be quicker to create a recursive call into the template it may not be known which parts of the spun documents are known. That method was certainly adequate for the ArticleBot kind of processing that existed earlier but there is another problem. Speed will be important really only when producing loads of output so parsing them first seems to be the best idea.

So the first pass into the template from beginning to end will be to pull out the codes or spun items into variables. These can hang around and get used when the article is being spun.

Let’s see what we can use…

First off here are the basic needs. Scan through an article, we stick to text for now, and pick up commands to allow different content to be chosen at specified points. For now the easiest and most well know is to use the characters ‘{‘ and ‘}’ to show the beginning and the end of the text to spin and a character ‘|’ to show where the splits will occur.

So easiest article to spin will be something like this:

{Hello|Hi} World

Two results will be either

Hello World

Or

Hi World

But we want it to be a bit more powerful than that don’t we. We want to be able to spin within a spin something like this:

{So that's {the end|finished} |that's it…}

Three results will be

So that’s the end

So that’s finished

That’s it…

The above is much more useful with longer phrases but that essentially shows that multiple levels of spin are necessary and I do not want any limits.

Right, more requirements, I said I wanted the template to do more, but in the absence of defining exactly what I want now I will need to consider global and local scripts. Another character is needed, let’s use ‘^’ and just in case we actually need ‘^’ in our output then make the characters ‘^^’ produce it.

The characters I am suggesting I will keep open and in a state where they can be changed at any time in the future. I want to be able to use this a lot before settling down to what the default characters should be.

Right, so how can we use scripts or fields with the ‘^’ character.

Let’s try this example, a light bulb page

Light type {^light_type} and { can be found | is available } in wattage of {^wattage_list}

To produce something like:

Light type is neon and is available in wattage of 10W,20W and 30W

Light type is LCD and can be found in wattage of 1W and  2W

I made an assumption here that the variables may be anything and kept the ‘{}’ as delimiters as well because I want to have complete control over spacing. So anything whitespace with these delimiters (maybe I will need to think of quotation marks} will be absorbed. So I can write my template like this:

Light type {

^light_type

} and {

can be found | is available

} in wattage of {^wattage_list}

Or however way I please.

We’ll see if that works or not.

There maybe optional bits in the template too. For example if a value is know then you might want to spin it but if not you do not want any blank values in the text.

Maybe something like

Light type {  ^light_type } {^if(wattage_list) and { can be found | is available } in wattage of {^wattage_list} }

Finally, to handle the scripting I need to choose the engine. In QKSite I used VBScript because of its simplicity but also allow JavaScript be used if necessary. Maybe those variable will be functions with arguments in parentheses, like in the ‘if’ above – we’ll see how it looks.

While I make that decision I will continue and build QKSite solely to produce the spinning text for version 1. It looks about as complicated as I would put up with and playing around with it might enable better ways to be imagined.

Let me know if you have any thoughts on this…

My aim is to keep the template creation as quick and easy as possible without getting untidy, I want to concentrate then on the articles and not any coding.