Debugging VBScript

This blog post only exists to prevent expired linking. My blog has moved to: http://pabloaizpiri.com/

--------------------------------------------------------------------------------

Unfortunately, sometimes I still have to work with VBScript. It actually isn't so bad for quickly whipping up a script that runs on a schedule to perform some sort of minor task. Sometimes it comes in handy to debug the script, and I'm making an entry on it for future reference. (A coworker showed me this)

1.) Make sure you set your VS settings as follows in Tools -> External Tools:

2.) Set your Internet Explorer settings accordingly:

3.) Load the script you wish to debug into Visual Studio... make sure it is VALID syntax or you'll never even hit the break point.

4.) Set your break points.

5.) Go to Tools -> VBSCript_Debugger to begin debugging.

NOTE: When you hit "stop", it will not actually stop script execution- only the debugging session- to actually stop the script from continuing, use the Immediate window to execute "wscript.quit".

You can also start the debugger on a server that does not have Visual Studio running by starting up a command prompt and firing up the VBScript debugger manually:

wscript.exe //d c:/test.vbs

You'll need to set a break point. (The command is "Stop" and you can add it anywhere in your script). This should automatically fire up the Microsoft Visual debugger if installed; if not you can download it from the Microsoft website.

As a final note, remember that often time errors in script are due to the account they are running under.  This one's caused me to spin my wheels for linger that I wanted to until I realized my scheduled task was running under credentials preventing it from completing part of its job... it's always the simple things. :P

What Can't be Measured Can't be Improved: Investing in Measurement

This blog post only exists to prevent expired linking. My blog has moved to: http://pabloaizpiri.com/

--------------------------------------------------------------------------------

For sure we have all heard the saying and know it well- but few of the times do we apply it- especially if measuring (or making measuring easy) is going to take some work up front. We like to think we can "cowboy" it and figure out where out performance-hitting bugs are ourselves.

Today, I worked on my 3D engine most of the day. I had a blast and a lot of good came from all the work I put into it. Even though I still have a ways to go, I felt like I had gotten to the point where it was time to start measuring performance.

Performance is a pretty important topic for a 3D application- and especially considering I'm just starting out learning the ins and outs of DirectX (and to add insult to injury, I'm using a managed language), I wanted to make sure I'm writing fairly efficient code. I've learned not to micro-optimize, and I felt like now that I had gotten the basic concepts down and working I would begin profiling.

At first I was going to just insert a few Diagnostics.Stopwatch calls here and there but after thinking it through I decided that as important as performance was for a 3D engine, it would most likely be in my best interest to create a class to make measuring the performance of the engine very easily. This is important because as you're working, (if the use of your diagnostics class is easy enough), you will also be encouraged to add metrics to your application. I've realized that the time spent up-front to do this is invaluable, especially since I know I will use this class plenty in the future.

My approach was fairly simple- I knew it had to be something that was super easy to use or it would have a low ROI, but I also knew it had to provide valuable performance metrics and information, not just data.
Essentially it has four major timing calls:

• public void BeginSingle()
• public void EndSingle(string Name)
• public void BeginSpecific(string Name) 
• public void EndSpecific(string Name)

BeginSingle and EndSingle make timing a certain portion of code very easy. For example, say I want to measure a piece of code that calculates what should be hidden in a scene:

            TimingObject.BeginSingle();
     [..code that calculates visibility ...]
     TimingObject.EndSingle("CalculatingHiddenObjects");

The string passed to "EndSingle" will ensure that value it attached to some identifier. Now, if I need to nest between timings, I can use BeginSpecific and EndSpecific:

            TimingObject.BeginSpecific("TotalTime");
     TimingObject.BeginSingle();
     [..code that calculates visibility ...]
     TimingObject.EndSingle("CalculatingHiddenObjects");
     TimingObject.BeginSingle();
     [..other stuff ...]
     TimingObject.EndSingle("OtherCalculations");
     TimingObject.EndSpecific("TotalTime");

This model has worked remarkably well. Additionally, built into the class I have code that stores the times in an array and averages them, as well as takes note of the max and min values. The class also has a function to output all values to a text file, and provide percentages based on the total frame and entity render times sorted by the most expensive- this was a very useful feature for me!

Results? I have improved my code to where it is over 5x faster! Huge difference! It came much more in line with what I was hoping for. I'm very happy, and I was able to make excellent use of my time while learning some great lessons on how to write my DirectX code more efficiently. In line with the 80/20 "rule", indeed, about 80% of my bottle neck was in about 20% of my code. In the most complex scene I tested, I went from about 120 FPS, to about 680! That's a 566% improvement after about 25 minutes of constantly optimizing by testing out different things and looking through the generated performance log.

In closing, I felt like this was a successful approach I want to repeat again in other areas: first I learned how to write code that accomplished my goals without having to think about performance and I didn't waste my time micro-optimizing. Then, when I was ready, I invested some time up front to write a tool that made generating, collecting, and reading metrics easily and used that to tackle optimization. The 30 minutes that I spent up-front writing my timing class was definitely worth it in the end, and I would dare say probably saved me plenty of time.

JDX Update: Limited Gile[s] support in Engine

This blog post only exists to prevent expired linking. My blog has moved to: http://pabloaizpiri.com/

--------------------------------------------------------------------------------

Another tool that I used to love working with was 3D Gile[s] . It's a light mapper, and you can use it to build 3D scenes and light them. So I started working on support to import 3D Gile[s] scenes. Even though I'm still working through bugs and this scene import is not 100% yet, here's a screen shot of my progress so far:

Also did some minor optimizations of the engine itself.  EDIT: [These minor optimizations were before the major optimizations I made by profiling]

3D Engine: DirectX, C++, C#, SlimDX, and SharpDX

This blog post only exists to prevent expired linking. My blog has moved to: http://pabloaizpiri.com/

--------------------------------------------------------------------------------

Programming Love & 3D Engines
I have an admission to make. My true love of programming comes from game development. Sure, I love a good web app- but honestly I just love cool technology. Networking is cool. Database internals are cool. But what is a cooler technology than a 3D Engine? now that's awesome!

3D Engine and games are intriguing to me because of the technical challenges that must be addressed. Most business application address some in one way or another, but usually 3D Engines have to address all of them and be very good. Sorting? Check. Searching? Check. Drawing? Check. Complex Math? Check. AI? Sound? Hardware? Ect.. Additionally, a good 3D engine has to not only implement complex algorithms, but requires a strong architecture to manage its complexity while remaining efficient and still allowing great power. It needs to be a database, (of triangles, essentially), constantly read input, perform all sorts of logic and all these things must come together seamlessly to create a great game.

How I Started
My first experience was with LEGOMINSTORMS when I was about 12, I programmed the RCX (2.0) for cool little game with a light sensor and a moving piece of paper. Next, GameMaker, when I was 13- then 3D RAD, A5 Game Studio Engine, and finally Blitz3D. Over the last couple years I've wanted to skip all the middle ware and write a 3D engine using OpenGL or DirectX. At one point I wrote a very simple software render, but that was about as close as it got... I always had a hard time picking up DirectX or OpenGL- it seemed like SO MUCH was thrown at you just to draw a triangle... and that was fine, except I wanted to know what all that code did. On top of that I'm not super proficient in C++ which most tutorials are in.

At Last: Beginning DirectX
One weekend while I was visiting a friend in Houston, he convinced me to buy an old book he knew I was interested in. (Read it the whole time we were at at Half Price Books). It was on writing a managed 3D engine- just what I wanted! I slowly started on it and I actually started making progress. However, it wasn't long before I realized I was working with Managed DirectX 9, a boat that had sunk and MS had abandoned- but not before I had made some good progress. Encouraged by the progress, I decided I'd push forward and try once again tackling DirectX through a managed interface or library like SlimDX. I had tried to use it in the past, but it had never worked out well, but in my search I also found SharpDX- which is essentially a library of "extern" C# calls to the DirectX API. I figured I might as well stay as close as possible, since most DirectX tutorials are for C++ which call the same API. Another major reason I chose it was because of SharpDX's performance.

This turned out to be pretty great. The DirectX API is actually starting to make a lot more sense now, and I feel I could also even jump straight into C++, however, for my needs performance is already overkill- and since I can develop much faster in C# with the familiarity of the .NET framework, I'm going to stick to that. I've actually have gone far enough where I've implemented a simple object management system (for moving entities around relatively) and built a import function for  MQO models. (Still limited though) MQO is the format for the 3D Modeller Metasequoia. I know it is strange and probably obscure, little-known modeler- but I found it a few years ago through the FMS website and have found it to be an absolutely EXCELLENT simple, easy to use, and free 3D modeller. I LOVE it!

Credit Where Credit Is Due
 I have to definitely give credit to the following sources for getting me where I am right now.

Introduction to 3D Game Engine Design Using DirectX 9 and C# - This was the book that helped to somehow "flip the switch" and help DirectX make sense after reading and coding only a couple chapters... (I skimmed through 3-4 of the others and never read the rest)
http://www.two-kings.de/ - Some help with the clear explanation/tutorials
http://zophusx.byethost11.com/tutorial.php?lan=dx9&num=0 - HUGE help. This guy does through DETAIL so that I could understand. Heavily considered switching to C++....
http://www.toymaker.info/Games/html/lighting.html - Helped my understanding of shaders.
http://www.rastertek.com/tutindex.html - Helped a LOT in understanding and writing shaders for multi-texturing and special effects.

And of course Wikipedia... (if you haven't donated, but you use it, you should!) and the DirectX Documentation. It's MUCH easier to read it now that I've grasped the major concepts though I undoubtedly have quite a bit to go. And finally thank God for the Internet and search engines...  if you're persistent you'll find what you need.

The JDX Engine
So that's my newest passion- building this managed 3D engine- I couldn't really think of a name and finally settled with "JDX". When I've made a fair amount of progress, I actually want to build a Recoil clone (I played this game when I was 13 and I have fond memories of it), since that would be easy to do and not require much artistic skills. If it turns out well, I'd love to also make this 3D engine public for anyone who wants to be able to write a managed DirectX 3D game without being a DirectX expert. It will probably be fashioned somewhat after Blitz3D API, since I've always found it to be extremely intuitive.

I'll probably give updates on JDX here and there when I can. Mostly I'm teaching myself, so I'm making a lot of mistakes. I'm very open to learning how to actually write an efficient and good D3D code base since a lot of examples *work* but there seems to be many different ways to do things in D3D and I want to not just perform the task but do so efficiently. Which brings me to one of my new major pet peeves: in most tutorials there seems to be very little out there as to how to actually best write the code. (E.g. do you store a vertex buffer for each object in your world or do you attempt storing them all in the same buffer? What about object parts? How do you apply the correct shaders when you do so? Ect...)

Until next time....

UPDATE: Bought another couple books from amazon on 3D Mathematics and DirectX 10; it's helping lots. Engine is coming along well... here are some updates.

NodeJS & Simple C# HTTP Server

This blog post only exists to prevent expired linking. My blog has moved to: http://pabloaizpiri.com/

--------------------------------------------------------------------------------

So... I've read quite a bit about NodeJS and was very early introduced to it by one of my good friends, Dominic ( http://dominicbarnes.us/ ). He's a huge JavaScript fan and so NodeJS was a big hit. I like Javascript, but admittedly I am nowhere near as proficient as he is nor do I understand fully the functional paradigm needed to use it to its full extent. (He's great at pretty much everything open source/other side of the MS/Windows fence, so if you need someone like that go hire him and pay him lots of money - you won't regret it.) NodeJS got attention with it's claims to efficiency with its non-blocking programming style- the functional JavaScript would in theory make writing such code easy.

The Bleeding-edge Event Model in NodeJS?
This was a couples months ago and I was fascinated by the potential NodeJS performance gains and decided I would try writing a simple server in C# with the same model. I figured since C# is compiled, it may be at the least somewhat quicker than the Windows version of NodeJS. Of course, now it all seems silly having learned what I did about IIS through the process. I never realized most of the comparisons for performance were against Apache, and IIS already performs better than Apache anyway. Of course, that seems obvious now- what was I expecting? I suppose at times we can all be suckers for the success stories of the underdog coming out on top, but in practice that is generally not the case. Regardless, it was a fun learning experience. My proof-of-concept server did turn out to be incredibly fast, (and considerably faster than the Windows NodeJS at the time) but that doesn't mean much considering it offered limited functionality. It was a simple test; I used .NET's HTTP Request classes and didn't build my own implementation to keep simple. (which would have been a huge part of the effort) It was really fun trying to think of ways to optimize my little server. (request handling/caching/reading from disk/ect.)

A C# Server Like NodeJS
Basically it is a C# application that only has a task bar tray icon for an interface (I never actually got far enough to turn it into a service and separate the UI from the server service) and sits around waiting for requests. A main thread is the one that just sits around listening to the HTTP port[s] all the time and whenever a requests comes through hands it off to a worker thread. The worker thread checks if the request is cached in memory and if so returns it, if not depending on the extension it will either return a static file, image, or compile the script page, add it to the cache, and return it. (or just run the script page's compiled code if cached)

Having one thread completely devoted to listening for requests and passing the actual request handling to a thread pool allowed my server to respond to requests extremely quickly- this was the concurrency I was after. The cached compiled scripts would run quickly once the compilation for the page script was cached, as it was literally like running a function that returned a string. (Plus obviously the JIT will also compile to native code once the function is called for the first time)

Yes, it compiles on demand! But that wasn't as big a deal as I thought it would be. It was much easier than I thought since .NET comes with compiler libraries for C#. All my server does is some string parsing on the requested file to look for '@{' and '}@' symbols to know where the C# code begins and ends. (C# "script", anyone?) As I mentioned, compiled methods are kept in memory so that subsequent requests are extremely fast.

I realize NodeJS operates a bit differently. NodeJS listens on a single thread (main event loop) and when a request comes in, it immediately processes it. For requests what would require "blocking" functions, (such as File I/O... though technically any function call is blocking by definition), a callback is given and the "blocking" function is queued in a thread pool- this way the main even loop thread goes right back to processing and listening to requests. When the "blocking" function completes, it calls the callback function on the main event loop and the request is finished on that main thread. (As I thought through this I began to realize some potential shortcomings)

In my case, I simply created the event loop to only listen for requests, and then hand off request handling to the thread pool. This is because I couldn't be guaranteed that the page script wouldn't perform a "blocking" operation. However, had I continued with the project and had my intention been to imitate NodeJS exactly, I would have probably needed to build a library of functions that page scripts could have called to handle "blocking" functions. This is where the power or ease of the functional programming style of JavaScript would have been nice. Doing this in C# would have been ugly, but it would be much easier to adopt Javascript's call-back functional style to help segregate those "blocking operations" that should be executed in the thread pool from those that should run in the main event loop. (Technically I didn't have a "main event loop" since all mine did was handle and hand off the requests, but you get the point) In the end, it didn't really matter that I didn't go through all the lengths to simulate that, because for my tests I wrote a page script that didn't do any file I/O or any other "blocking" operations. (Which I suppose makes for poor tests, but good enough for what I needed)

Realizations and Some Final Thoughts on NodeJS & IIS
Throughout the whole time I was researching more about the IIS pipeline and began to realize IIS does pretty much the same thing as my server did as far as listening, handling off requests, and working with a thread pool to process them. (of course, it does it a whole lot better) Eventually I stopped development; the code is here if curious. (Since I abandoned the project, the "scripting" support is very limited- it supports C# since it uses the .NET compiler but the page script code doesn't have access to any server variables like POST/GET, ect making it close to useless)

So NodeJS/IIS thoughts. I think it is a cool technology and I've still got a lot to learn on the subject, but I've researched enough where I feel I have a fair opinion. I think IIS does a pretty darn good job and NodeJS model isn't exactly ground-breaking here... it's actually been around for a long time. Apache is the big web server it always seems to be compared to, and I suppose that's where there's a big win performance-wise is since Apache spawns a new thread for every request. (Maybe the just need to implement a thread pool in their pipeline?) My thoughts are you'd be hard pressed to get test results (and not just mass concurrent request tests) where an equivalent MVC.NET page written well and using IIS under-performs it's equivalent NodeJS page.

In closing, here are a couple articles I agree with, though I think he is rather harsh/offensive to the NodeJS community, but he seems to hit the nail on the head as far as analyzing performance and the NodeJS model:
http://teddziuba.com/2011/10/node-js-is-cancer.html
http://teddziuba.com/2011/10/straight-talk-on-event-loops.html

Part 2: SqlBulkCopy Class (MS SqlServer and .NET)

This blog post only exists to prevent expired linking. My blog has moved to: http://pabloaizpiri.com/

--------------------------------------------------------------------------------

At work we're making an effort to contribute technical knowledge to a centralized IT wiki. I like that. Writing encouraged a good understanding of the technology and it's benefits. I didn't think I'd have much to write but I'm surprised how some things I take for granted as simple, others didn't know and vice versa. This two part series is from those entries.

​Introduction

Normally, inserting rows into SQL server is quick and easy and done through a simple INSERT SQL statement. This is fine when saving data to one, two, or even a few rows. However, when it is necessary to insert larger sets of data, this method becomes not only functionally inadequate, but slow and clunky. In this entry (part two of a two-part series) I wanted to write about the second option we will look at for inserting large sets of data: using .NET’s SQLBulkCopy class.

The SqlBulkCopy ClassWhen it is necessary to insert more than about a 1000 rows of data, a TVP​ now begins to reach the limits of its performance gain. If we are using the TVP’s only for inserts, we can move up and dramatically increase performance by using .NET’s SqlBulkCopy class. In addition to providing the functionally for large inserts, the SQLBulkCopy class can also be used to copy large amounts of data between tables. With SQLBulkCopy we can deal with millions of rows if need be. Here is an amazing whitepaper on the SqlBulkCopy class’ performance:http://www.sqlbi.com/LinkClick.aspx?fileticket=svahq1Mpp9A%3d&tabid=169&mid=375

The SqlBulkCopy Class
Using the SQLBulkCopy class if fairly simple. The dataset you use must match the columns on the table. If the order or column names are a bit different, that’s okay since that can be handled with the SqlBulkCopy class’ ColumnMapping property which is just for that. Here’s a .NET sample of using the SqlBulkCopy class to update a table named “tblSIWellList” from a table name “MyData” within a DataSet:

Using objConnection As System.Data.SqlClient.SqlConnection = GetSQLConnection()

    objConnection.Open()

           

    Using bulkCopy As SqlBulkCopy = New SqlBulkCopy(objConnection)

        bulkCopy.DestinationTableName = "dbo.tblSIWellList"

        With bulkCopy.ColumnMappings

            .Add(New SqlBulkCopyColumnMapping("ID", "ID"))

            .Add(New SqlBulkCopyColumnMapping("EPD", "EPDate"))

            .Add(New SqlBulkCopyColumnMapping("Comments", "Comments"))

            .Add(New SqlBulkCopyColumnMapping("Date", "Date"))

            .Add(New SqlBulkCopyColumnMapping("RTP_Date", "RTPDate"))

        End With

        bulkCopy.WriteToServer(ds.Tables("MyData"))

    End Using

End Using

Because SqlBulkCopy class is designed to copy/insert a large number of rows, transactions are handled in batches. It is possible to specify how large each batch is (e.g. 5000 rows at a time) but by default a single transaction (“batch”) is used for all rows. When committing transaction in batches, a failed batch will only roll back the last active transaction in the batch. (This may not necessarily be all rows if a previous batch was successfully committed)

Considerations when using .NET’s SlqBulkCopy class

There are a few “gotcha”’s to keep in mind when using the SqlBulkCopy class:

• When the source and destination table data types are different, SqlBulkCopy will attempt to convert to the destination data type where possible but this will incur a performance hit.
• By default, PK’s are assigned by destination and are not preserved.
• By default, constraints are not checked and triggers are not fired. Also row-level locks are used. Changing these setting may affect performance.​

Part 1: Table Valued Parameters (SQL Server 2008+)

This blog post only exists to prevent expired linking. My blog has moved to: http://pabloaizpiri.com/

--------------------------------------------------------------------------------

At work we're making an effort to contribute technical knowledge to a centralized IT wiki. I like that. Writing encouraged a good understanding of the technology and it's benefits. I didn't think I'd have much to write but I'm surprised how some things I take for granted as simple, others didn't know and vice versa. This two part series is from those entries.


Introduction
Normally, inserting rows into SQL server is quick and easy and done through a simple INSERT SQL statement. This is fine when saving data to one, two, or even a few rows. However, when it is necessary to insert larger sets of data, an array, or some kind of dynamic list atomically, this method becomes not only functionally inadequate, but slow and clunky. This may also be the case when updating a large number of records individually. In this entry (part one of a two-part series) I wanted to write about the first of two options we will look at for inserting/updating larger sets of data: Table Valued Parameters.

  

Using Table Valued Parameters in SQL Server 2008
When it is necessary to insert or update more than just a few rows or a list of data, a TVP (table valued parameter) is the next step. TVP’s became available in SQL Server 2008 and are perfect for sending a stored procedure a dataset of anywhere from a few rows to around a thousand.

To demonstrate, I am going to use some samples from a recent project that I updated to use TVPs. In order to create a stored procedure that can take a table as a parameter, first we create a UDT (User Defined Type) of the table type. You create your user defined table type (UDTT) just like you would normally create a SQL server table:

CREATE TYPE [dbo].[typProjectStreamLight] AS TABLE(
[PCID] [int] NOT NULL,
[Stream] [varchar](100) NOT NULL,
[StreamType] [varchar](20) NOT NULL,
[BudgetMonthLabel] [varchar](20) NOT NULL,
[Value] [decimal](18, 6) NULL
)

GO

Once you have created the UDTT, you can create a stored procedure that will take the UDT as a parameter:

-- =============================================
CREATE PROCEDURE [dbo].[spProjectStreams_AllMonths_set]
@CID INT,
@UserName VARCHAR(25),
@tvpMonthValueList typProjectStreamLight READONLY
AS
BEGIN

 In this example, the parameter is named “@tvpMonthValueList” and you can see the UDT as the declared data type next to the parameter name. (“typProjectStreamLight”) Note that to use UDTT’s as arguments you must declare them as read-only.

Here’s another sample using TVP’s as a way to emulate a list of data or an array. Here’s the UDTT declaration:

CREATE TYPE [dbo].[typIntList] AS TABLE(
[Value] [int] NOT NULL
)
GO

And here the procedure using it as an argument:

CREATE PROCEDURE [dbo].[spUpdateCalculatedStreamsFromList]
@PCIDList [typIntList] READONLY,
@LastUpdatedBy VARCHAR(25)
AS
BEGIN


Communicating between .NET and SQL Server 2008 using TVPs
Using a stored procedure with .NET is very simple- the main difference is a SQL Data Type of “Structured” is assigned to the parameter containing the table. Because the System.Data.SqlClient supports populating table-valued parameters from DataTable, DbDataReader or IEnumerable objects, populating the UDTT is very easy. Below is an example with the generic setup for the SQL command same as always plus our new table:

Dim objConnection As SqlConnection = DataAccess.modConnection.GetSQLConnection
objConnection.Open()
Using objCommand As SqlCommand = objConnection.CreateCommand()
With objCommand
.CommandType = CommandType.StoredProcedure
.CommandText = "spProjectStreams_AllMonths_set"
.Parameters.AddWithValue("@CID", CID)
.Parameters.AddWithValue("@UserName", DBNullIF(UserName))
.Parameters.AddWithValue("@tvpMonthValueList", mrshlTable)
.Parameters("@tvpMonthValueList").SqlDbType = SqlDbType.Structured.ExecuteNonQuery()
End With
End Using

In the sample above, “mrshTable” is a DataTable object that matched the schema of the “typProjectStreamLight” UDTT described previously.

Considerations when using TVP’s
There are a couple of things to keep in mind when using UDTT’s: (At least as of July 2011)

• As noted above, UDTTs must be declared as read-only when used as an argument in a stored procedure- this means you won’t be able to modify the data in the UDTT.
• UDTT’s cannot be passed to a SQL function.
• UDTT’s cannot be modified after created- they must be dropped and created again.
• UDTT’s cannot be dropped if there are any stored procedures currently using them as arguments. First the stored procedure must be dropped or altered to not use the UDTT and then the UDTT may be dropped.

Here are some benefits you may reap by using TVP’s correctly:

• Update/Insert operations can be performed with less network overhead.
• Using a UDTT in a stored procedure allows performing your INSERT/UPDATE operations in a single set allowing for a faster, more efficient SQL Server performance
• By using a stored procedure that accepts a UDTT, using one connection, running the whole operation in a transaction, and staying away from ad-hoc SQL become much easier and the default.
• I’ve found it may be much easier to write/refactor code in .NET to use a table and simply pass it to the stored procedure when we are ready to insert/update the data.