tag:blogger.com,1999:blog-71764099626791911962024-03-13T08:53:04.009-07:00Jhott EngineeringArticles, Solutions, Workarounds, and thoughts on technology in general. A programmer's blog, by no means authoritative.The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.comBlogger18125tag:blogger.com,1999:blog-7176409962679191196.post-21513031447611625962015-01-31T11:49:00.000-08:002015-08-23T16:15:13.281-07:003D Engine Update<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
So my pet project of writing a simple 3D engine basically was halted ever since my last "JDX Engine" post.... I've been busy; I got married, received a promotion, moved, and bought a house- so it's been crazy. However, I think a big part of it was also that I let it stagnate and never came back to it. I also know I'm reinventing the wheel writing these kinds of things, and even though it's mostly a project to learn how to deal with the challenges in the 3D arena, it can be a bit demotivating when it takes a long time to really have anything of substance to demo when you can just pick up something like <a href="http://unity3d.com/">Unity</a>.<br />
<div>
<br /></div>
<div>
Having said that, I will always find 3D engines and their challenges intriguing. I'd love to eventually build a simple game with a 3D engine of my own design. I ended up deciding to try again now that I'm more settled and I wrote another 3D engine, mostly through the help of <a href="http://www.amazon.com/Introduction-3D-Game-Programming-DirectX/dp/1936420228">Frank Luna's "3D Game Programming with DirectX 11" book</a>. In spirit, it's the successor to "JDX" in that it's in C# and uses Slim DX.
</div>
<div>
<br /></div>
<div>
I realize there's a good chance this will be a lot like my last 3D engine project- I'll make some progress and then I'll get bogged down with something and it may not get worked on for months. Maybe I'll never even touch it again. However, I enjoy building this kind of stuff, and I'm posting my code on <a href="https://github.com/JhottMaster/3DEngineTest">GitHub</a>, so maybe it will serve to help someone get started on their journey. Also open to feedback on it.
</div>
<div>
<br /></div>
<div>
A lot like last time, it's pretty simple. I don't have support for the 3D Giles format like the previous engine, but I can't even seem to find that application anymore so that's probably for the better. I would like to find a good light mapper and then write support to import whatever its native format is so I have something that can start looking like a game. I'm thinking BSP seems most common but I haven't decided yet.
</div>
<div>
<br /></div>
<div>
Here's what I <i>do </i>have:
</div>
<div>
<ul>
<li>DirectX 11 via SlimDX - most modern DirectX API.</li>
<li>Support for up to (currently) 8 dynamic lights - spot, directional, or point.</li>
<li>New framework of classes that should make using available objects much more simpler and easier.</li>
<li>Although still not full, much better support for MQO files. </li>
<ul>
<li>Previous attempt only loaded triangulated meshed, this loads quads as well so you don't have to do anything special to your mesh in Metasequoia.</li>
<li>Support for mirroring on the X-axis.</li>
<li>Loads all Metasequoia objects individually and can organize them by material type for faster rendering. (The engine doesn't take advantage of this yet)</li>
</ul>
</ul>
<div>
Here's a few screenshot showcasing what I have so far.
</div>
</div>
<div>
<br /></div>
<div>
Further support for MQO files:
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table>
<tbody>
<tr>
<td><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;">
<tbody>
<tr>
<td style="text-align: center;"><a href="http://4.bp.blogspot.com/-mCwaBJL-wS4/VM0rDbxjFRI/AAAAAAAAHXo/7AsUj7vs35I/s1600/violin.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="250" src="http://4.bp.blogspot.com/-mCwaBJL-wS4/VM0rDbxjFRI/AAAAAAAAHXo/7AsUj7vs35I/s1600/violin.jpg" width="320" /></a></td>
</tr>
<tr>
<td class="tr-caption" style="text-align: center;">Metasequoia's sample "violin" (with Z-axis removed)</td>
</tr>
</tbody>
</table>
</td>
<td><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;">
<tbody>
<tr>
<td style="text-align: center;"><a href="http://1.bp.blogspot.com/-lZV4PVWW6NI/VM0psm4rvHI/AAAAAAAAHXg/U4Wh8fVVR_Q/s1600/Heli.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="250" src="http://1.bp.blogspot.com/-lZV4PVWW6NI/VM0psm4rvHI/AAAAAAAAHXg/U4Wh8fVVR_Q/s1600/Heli.jpg" width="320" /></a></td>
</tr>
<tr>
<td class="tr-caption" style="text-align: center;">AH-64 Apache model</td>
</tr>
</tbody>
</table>
</td>
</tr>
<tr>
<td><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;">
<tbody>
<tr>
<td style="text-align: center;"><a href="http://4.bp.blogspot.com/-e44z1ejXdPU/VM0ttNXWK9I/AAAAAAAAHX0/os3-esaBEoU/s1600/skull.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="250" src="http://4.bp.blogspot.com/-e44z1ejXdPU/VM0ttNXWK9I/AAAAAAAAHX0/os3-esaBEoU/s1600/skull.jpg" width="320" /></a></td>
</tr>
<tr>
<td class="tr-caption" style="text-align: center;">1 dynamic spot light; resource model from book.</td>
</tr>
</tbody>
</table>
</td>
<td><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;">
<tbody>
<tr>
<td style="text-align: center;"><a href="http://1.bp.blogspot.com/-v97R0kYp8lg/VM0tu0yHY2I/AAAAAAAAHX8/VX9yfJ9k3Ho/s1600/skullLighting.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="232" src="http://1.bp.blogspot.com/-v97R0kYp8lg/VM0tu0yHY2I/AAAAAAAAHX8/VX9yfJ9k3Ho/s1600/skullLighting.jpg" width="320" /></a></td>
</tr>
<tr>
<td class="tr-caption" style="text-align: center;">3 dynamic spot lights</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: left;">
I think the main things that I need to add support for next are:
</div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<ul>
<li>Sprites</li>
<li>Multi-textures.</li>
<li>Support to load some common level/map format that includes light maps</li>
<li>Optimization in rendering - this one will take a lot of work and would constantly be on-going.</li>
<ul>
<li>Minimizing calls to set materials/textures</li>
<li>Scene graphs</li>
<li>Frustum Culling</li>
</ul>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div>
<br /></div>
The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com1tag:blogger.com,1999:blog-7176409962679191196.post-13407642313004708242012-03-20T12:39:00.002-07:002015-08-23T16:15:25.351-07:00ABCpdf for Reporting and Rendering Table Borders<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">ABCpdf Awesomeness and Reporting</span><br />
<span style="font-size: large;"><br /></span>I love using <a href="http://www.websupergoo.com/abcpdf-1.htm">ABCpdf</a> when working on a project that needs to generate PDF files. It is very powerful and flexible. One of the main reasons I love, have used, and convinced companies I work for to purchase ABCpdf is because it can render a PDF file from HTML- and not just as a big PDF image either.<br />
<br />
Anway, the idea of HTML to PDF is extremely powerful- often times rather than use a reporting package that requires tons of work to modify a report to the exact custom request of a customer, I would much rather generate an HTML page and convert it to PDF or whatever reporting format is requried. (PDF seems to usually be the most popular) This is has become invaluable, as editing or creating new HTML pages that pull data is easy, straightforward, quick and powerful, and with the ability tocovert the output to PDF I can now send the PDFs as reports or have them available for download. I've frequently avoided nightmares when updating or adding to reports, and I've been able to modify reports very quickly using this model.<br />
<br />
<br />
<span style="font-size: large;">Rendering HTML Table Borders with ABCpdf</span><br />
<br />
Recently I found a "bug" of sorts when rendering an HTML table border using ABCpdf. It's a bit of an odd bug. Essentially the borders become different thickness and really distract from the table. Since reports often include tables, having tables show up correclty and properly formatted is crucial- after all, this is one of the main reasons I like to generate reports in HTML and then convert to PDF: aesthetic flexibility.<br />
<div class="separator" style="clear: both; text-align: -webkit-auto;">
<br /></div>
A picture is worth a thousand words- here is a screenshot of the browser's rendering of a table border:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-5R22Cyk2604/T2jY0y6aETI/AAAAAAAAAI0/JBuJqGgDgd8/s1600/HTMLRender.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" src="http://3.bp.blogspot.com/-5R22Cyk2604/T2jY0y6aETI/AAAAAAAAAI0/JBuJqGgDgd8/s1600/HTMLRender.jpg" /></a></div>
<br />
<br />
That's perfect. Just what I want. Now convert it to PDF:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-N3PlWobmVaM/T2jZSS5-KLI/AAAAAAAAAI8/1YnWVMm1Kw0/s1600/PDFRender.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-N3PlWobmVaM/T2jZSS5-KLI/AAAAAAAAAI8/1YnWVMm1Kw0/s1600/PDFRender.jpg" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Ouch... whatever happened to those table borders? It's odd- because if you zoom in on the PDF borders show up fine: </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-Z8vMyjMdnZQ/T2jZoPs0I4I/AAAAAAAAAJE/FjAyR0LtNHY/s1600/PDFZoom.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="http://3.bp.blogspot.com/-Z8vMyjMdnZQ/T2jZoPs0I4I/AAAAAAAAAJE/FjAyR0LtNHY/s320/PDFZoom.jpg" width="117" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
At first I thought it might be due to not enough pixel density, and that maybe I just needed to increase the "browser width" and image resolution settings on ABCpdf so I would have a sharper image to sample from. Then I realized ABCpdf is going to try to parse the table, since it's not an image, so had to be a CSS issue. Obviously the borders were being created in some way that shows the right size at a certain zoom but is slightly off- enough to show when you use a fit-to-screen size.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Sure enough it was- and the fix was rather simple. I had a 'style="border 1px solid black"' on the table tag, and then the cells within the table also included inline styling data. For some reason ABCpdf is processing these borders that are defined twice- (first in the table as a whole and then in the cells as necessary)- as "overlapping". Removing the inline styling information from the table tag and leaving it on the cells corrected the issue:</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-0tviSTBC9u4/T2jamOfzVVI/AAAAAAAAAJM/GVhgCXJVx14/s1600/PDFRenderCorrect.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-0tviSTBC9u4/T2jamOfzVVI/AAAAAAAAAJM/GVhgCXJVx14/s1600/PDFRenderCorrect.jpg" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Yes, yes, I shouldn't be using "inline" styles AND potentially this would not even have been a problem with a table-less div layout, but I was working with what I had and I wasn't about to rewrite the page if there was a reasonable quick fix. I did however, remove most of the inline styles and replace them with CSS classes... you do what you can. :)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<br />
<br />
<br />The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com1tag:blogger.com,1999:blog-7176409962679191196.post-67748228250113099292011-12-19T09:28:00.000-08:002015-08-23T16:17:02.990-07:00Debugging VBScript<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
Unfortunately, sometimes I still have to work with VBScript. It actually isn't so bad for quickly whipping up a script that runs on a schedule to perform some sort of minor task. Sometimes it comes in handy to debug the script, and I'm making an entry on it for future reference. (A coworker showed me this)<br />
<br />
<br />
<div style="background-color: white; line-height: 18px; margin-bottom: 1em;">
<span class="Apple-style-span" style="font-family: inherit;">1.) Make sure you set your VS settings as follows in Tools -> External Tools:</span></div>
<div style="background-color: white; line-height: 18px; margin-bottom: 1em;">
<span class="Apple-style-span" style="font-family: inherit;"><img alt="VSSettings.jpg" src="http://nfxtulsp2010/IT/Blog/Lists/Photos/VSETSettings.jpg" style="margin-bottom: 5px; margin-left: 5px; margin-right: 5px; margin-top: 5px;" /></span></div>
<div style="background-color: white; line-height: 18px; margin-bottom: 1em;">
<span class="Apple-style-span" style="font-family: inherit;">2.) Set your Internet Explorer settings accordingly:</span></div>
<div style="background-color: white; line-height: 18px; margin-bottom: 1em;">
<span class="Apple-style-span" style="font-family: inherit;"><img alt="IESettings.jpg" src="http://nfxtulsp2010/IT/Blog/Lists/Photos/IESettings.jpg" style="margin-bottom: 5px; margin-left: 5px; margin-right: 5px; margin-top: 5px;" /><br /><br />3.) Load the script you wish to debug into Visual Studio... make sure it is VALID syntax or you'll never even hit the break point.</span></div>
<div style="background-color: white; line-height: 18px; margin-bottom: 1em;">
<span class="Apple-style-span" style="font-family: inherit;">4.) Set your break points.</span></div>
<div style="background-color: white; line-height: 18px; margin-bottom: 1em;">
<span class="Apple-style-span" style="font-family: inherit;">5.) Go to Tools -> VBSCript_Debugger to begin debugging.</span></div>
<div style="background-color: white; line-height: 18px; margin-bottom: 1em;">
<span class="Apple-style-span" style="font-family: inherit;"><b>NOTE: When you hit "stop", it will not actually stop script execution- only the debugging session- to actually stop the script from continuing, use the Immediate window to execute "wscript.quit".</b></span></div>
<div style="background-color: white; line-height: 18px; margin-bottom: 1em;">
<span class="Apple-style-span" style="font-family: inherit;">You can also start the debugger on a server that does not have Visual Studio running by starting up a command prompt and firing up the VBScript debugger manually:</span></div>
<div style="background-color: white; line-height: 18px; margin-bottom: 1em;">
<span class="Apple-style-span" style="font-family: inherit;">wscript.exe //d c:/test.vbs</span></div>
<div style="background-color: white; margin-bottom: 1em;">
<span class="Apple-style-span" style="line-height: 18px;">You'll need to set a break point. (The command is "Stop" and you can add it anywhere in your script). This should automatically fire up the Microsoft Visual debugger if installed; if not you can download it from the Microsoft website.</span></div>
<div style="background-color: white; margin-bottom: 1em;">
<span class="Apple-style-span" style="line-height: 18px;">As a final note, remember that often time errors in script are due to the account they are running under. This one's caused me to spin my wheels for linger that I wanted to until I realized my scheduled task was running under credentials preventing it from completing part of its job... it's always the simple things. :P</span></div>
The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-82043917046525912992011-12-17T18:33:00.000-08:002015-08-23T16:18:16.986-07:00What Can't be Measured Can't be Improved: Investing in Measurement<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
For sure we have all heard the saying and know it well- but few of the times do we apply it- especially if measuring (or making measuring easy) is going to take some work up front. We like to think we can "cowboy" it and figure out where out performance-hitting bugs are ourselves.<br />
<br />
Today, I worked on my 3D engine most of the day. I had a blast and <a href="http://jhottengineering.blogspot.com/2011/12/limited-giles-support-in-jdx-engine.html">a lot of good came from all the work I put into it</a>. Even though I still have a ways to go, I felt like I had gotten to the point where it was time to start measuring performance.<br />
<br />
Performance is a pretty important topic for a 3D application- and especially considering <a href="http://jhottengineering.blogspot.com/2011/11/3d-engine-directx-c-c-slimdx-and.html">I'm just starting out learning the ins and outs of DirectX</a> (and to add insult to injury, I'm using a managed language), I wanted to make sure I'm writing fairly efficient code. I've learned <a href="http://www.codinghorror.com/blog/2005/01/the-real-cost-of-performance.html">not to micro-optimize</a>, and I felt like now that I had gotten the basic concepts down and working I would begin profiling.<br />
<br />
At first I was going to just insert a few Diagnostics.Stopwatch calls here and there but after thinking it through I decided that as important as performance was for a 3D engine, it would most likely be in my best interest to create a class to make measuring the performance of the engine very easily. This is important because as you're working, (if the use of your diagnostics class is easy enough), you will also be encouraged to add metrics to your application. I've realized that the time spent up-front to do this is invaluable, especially since I know I will use this class plenty in the future.<br />
<br />
My approach was fairly simple- I knew it had to be something that was super easy to use or it would have a low ROI, but I also knew it had to provide valuable performance metrics and information, not just data.<br />
Essentially it has four major timing calls:<br />
<br />
<ul>
<li><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: blue;">public<span class="Apple-style-span" style="background-color: white;"> </span></span><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: blue;">void </span></span>BeginSingle()</span></li>
<li><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: blue;">public void</span> EndSingle(<span class="Apple-style-span" style="color: blue;">string </span>Name)</span></li>
<li><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="color: blue;">public void </span>BeginSpecific(<span class="Apple-style-span" style="color: blue;">string </span>Name) </span></li>
<li><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-style-span" style="background-color: white;"><span class="Apple-style-span" style="color: blue;">public void </span></span>EndSpecific(<span class="Apple-style-span" style="color: blue;">string </span>Name)</span></li>
</ul>
<br />
BeginSingle and EndSingle make timing a certain portion of code very easy. For example, say I want to measure a piece of code that calculates what should be hidden in a scene:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">TimingObject.BeginSingle();</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> <i>[..code that calculates visibility ...]</i></span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> TimingObject.EndSingle(<span class="Apple-style-span" style="color: #990000;">"CalculatingHiddenObjects"</span>);</span><br />
<br />
The string passed to "EndSingle" will ensure that value it attached to some identifier. Now, if I need to nest between timings, I can use BeginSpecific and EndSpecific:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">TimingObject.BeginSpecific(<span class="Apple-style-span" style="color: #990000;">"TotalTime"</span>);</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> TimingObject.BeginSingle();</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span><i><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span><span class="Apple-style-span" style="font-family: inherit;">[..code that calculates visibility ...]</span></i><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> TimingObject.EndSingle(<span class="Apple-style-span" style="color: #990000;">"CalculatingHiddenObjects"</span>);</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> TimingObject.BeginSingle();</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> </span><i><span class="Apple-style-span" style="font-family: inherit;">[..other stuff ...]</span></i><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> TimingObject.EndSingle(<span class="Apple-style-span" style="color: #990000;">"OtherCalculations"</span>);</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> TimingObject.EndSpecific(<span class="Apple-style-span" style="color: #990000;">"TotalTime"</span>);</span><br />
<br />
This model has worked remarkably well. Additionally, built into the class I have code that stores the times in an array and averages them, as well as takes note of the max and min values. The class also has a function to output all values to a text file, and provide percentages based on the total frame and entity render times sorted by the most expensive- this was a very useful feature for me!<br />
<br />
Results? I have improved my code to where it is over 5x faster! Huge difference! It came much more in line with what I was hoping for. I'm very happy, and I was able to make excellent use of my time while learning some great lessons on how to write my DirectX code more efficiently. In line with the 80/20 "rule", indeed, about 80% of my bottle neck was in about 20% of my code. In the most complex scene I tested, I went from about 120 FPS, to about 680! That's a 566% improvement after about 25 minutes of constantly optimizing by testing out different things and looking through the generated performance log.<br />
<br />
In closing, I felt like this was a successful approach I want to repeat again in other areas: first I learned how to write code that accomplished my goals without having to think about performance <i>and</i> I didn't waste my time <a href="http://www.codinghorror.com/blog/2009/01/the-sad-tragedy-of-micro-optimization-theater.html">micro-optimizing</a>. Then, when I was ready, I invested some time up front to write a tool that made generating, collecting, and reading metrics easily and used that to tackle optimization. The 30 minutes that I spent up-front writing my timing class was definitely worth it in the end, and I would dare say probably saved me plenty of time.<br />
<br />
<br />The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-56501753912669287892011-12-17T13:05:00.000-08:002015-08-23T16:18:22.514-07:00JDX Update: Limited Gile[s] support in Engine<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
Another tool that I used to love working with was <a href="http://www.frecle.net/index.php?show=giles.screenshots">3D Gile[s]</a> . It's a light mapper, and you can use it to build 3D scenes and light them. So I started working on support to import 3D Gile[s] scenes. Even though I'm still working through bugs and this scene import is not 100% yet, here's a screen shot of my progress so far:<br />
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="http://4.bp.blogspot.com/-eHX4LcrKmcU/Tu0EF5vrZpI/AAAAAAAAAGw/4FrvLWxmBMU/s1600/GilesLM.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="250" src="http://4.bp.blogspot.com/-eHX4LcrKmcU/Tu0EF5vrZpI/AAAAAAAAAGw/4FrvLWxmBMU/s320/GilesLM.jpg" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="http://3.bp.blogspot.com/-as6sWYwl1Y4/Tu0M6FjpSCI/AAAAAAAAAG4/QQF2JESotBU/s1600/GilesLM2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="250" src="http://3.bp.blogspot.com/-as6sWYwl1Y4/Tu0M6FjpSCI/AAAAAAAAAG4/QQF2JESotBU/s320/GilesLM2.jpg" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="http://2.bp.blogspot.com/-QhzO3Megqi0/Tu0Ql20gpEI/AAAAAAAAAHA/OOq1CBNamtw/s1600/GilesLM3.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="250" src="http://2.bp.blogspot.com/-QhzO3Megqi0/Tu0Ql20gpEI/AAAAAAAAAHA/OOq1CBNamtw/s320/GilesLM3.jpg" width="320" /></a></div>
<br />
<br />
<div style="text-align: left;">
Also did some minor optimizations of the engine itself. <i> EDIT: [These minor optimizations were before <a href="http://jhottengineering.blogspot.com/2011/12/what-cant-be-measured-cant-be-improved.html">the major optimizations I made by profiling</a>]</i></div>
The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-63361405902922696692011-11-30T11:44:00.000-08:002015-08-23T16:18:30.415-07:003D Engine: DirectX, C++, C#, SlimDX, and SharpDX<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Programming Love & 3D Engines</span><br />
I have an admission to make. My true love of programming comes from game development. Sure, I love a good web app- but honestly I just <i>love cool technology. </i>Networking is cool. Database internals are cool. But what is a cooler technology than a <i>3D Engine? </i>now that's <i>awesome!</i><br />
<br />
3D Engine and games are intriguing to me because of the technical challenges that must be addressed. Most business application address some in one way or another, but usually 3D Engines have to address all of them and be very good. Sorting? Check. Searching? Check. Drawing? Check. Complex Math? Check. AI? Sound? Hardware? Ect.. Additionally, a good 3D engine has to not only implement complex algorithms, but requires a strong architecture to manage its complexity while remaining efficient and still allowing great power. It needs to be a database, (of triangles, essentially), constantly read input, perform all sorts of logic and all these things must come together seamlessly to create a great game.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">How I Started</span><br />
My first experience was with <a href="http://mindstorms.lego.com/en-us/Default.aspx">LEGOMINSTORMS </a>when I was about 12, I programmed the RCX (2.0) for cool little game with a light sensor and a moving piece of paper. Next, <a href="http://www.yoyogames.com/make">GameMaker</a>, when I was 13- then <a href="http://www.3drad.com/">3D RAD</a>, <a href="http://www.3dgamestudio.com/">A5 Game Studio Engine</a>, and finally <a href="http://www.blitzbasic.com/Products/blitz3d.php">Blitz3D</a>. Over the last couple years I've wanted to skip all the middle ware and write a 3D engine using OpenGL or DirectX. At one point I wrote a very simple software render, but that was about as close as it got... I always had a hard time picking up DirectX or OpenGL- it seemed like SO MUCH was thrown at you just to draw a triangle... and that was fine, except I wanted to know what all that code did. On top of that I'm not super proficient in C++ which most tutorials are in.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">At Last: Beginning DirectX</span><br />
One weekend while I was visiting a friend in Houston, he convinced me to buy an <a href="http://www.amazon.com/Introduction-Engine-Design-Using-DirectX/dp/1590590813/ref=sr_1_1?s=books&ie=UTF8&qid=1322679234&sr=1-1">old book</a> he knew I was interested in. (Read it the whole time we were at at Half Price Books). It was on writing a managed 3D engine- just what I wanted! I slowly started on it and I actually started making progress. However, it wasn't long before I realized I was working with <a href="http://en.wikipedia.org/wiki/Managed_DirectX">Managed DirectX 9</a>, a boat that had sunk and MS had abandoned- but not before I had made some good progress. Encouraged by the progress, I decided I'd push forward and try once again tackling DirectX through a managed interface or library like <a href="http://slimdx.org/">SlimDX</a>. I had tried to use it in the past, but it had never worked out well, but in my search I also found <a href="http://code.google.com/p/sharpdx/">SharpDX</a>- which is essentially a library of "extern" C# calls to the DirectX API. I figured I might as well stay as close as possible, since most DirectX tutorials are for C++ which call the same API. Another major reason I chose it was because of <a href="http://code4k.blogspot.com/2011/03/benchmarking-cnet-direct3d-11-apis-vs.html">SharpDX's performance</a>.<br />
<br />
This turned out to be pretty great. The DirectX API is actually starting to make a lot more sense now, and I feel I could also even jump straight into C++, however, for my needs performance is already overkill- and since I can develop much faster in C# with the familiarity of the .NET framework, I'm going to stick to that. I've actually have gone far enough where I've implemented a simple object management system (for moving entities around relatively) and built a import function for MQO models. (Still limited though) MQO is the format for the 3D Modeller <a href="http://www.metaseq.net/english/">Metasequoia</a>. I know it is strange and probably obscure, little-known modeler- but I found it a few years ago through the <a href="http://n.ethz.ch/~mmoeller/fms/index_e.html"><span id="goog_1042341326"></span>FMS website</a> and have found it to be an absolutely EXCELLENT simple, <a href="https://www.google.com/search?rlz=1C1CHKZ_enUS436US436&gcx=w&sourceid=chrome&ie=UTF-8&q=Metasequoia+tutorial">easy to use</a>, and free 3D modeller. I LOVE it!<br />
<br />
<span class="Apple-style-span" style="font-size: large;">Credit Where Credit Is Due</span><br />
I have to definitely give credit to the following sources for getting me where I am right now.<br />
<br />
<a href="http://www.amazon.com/Introduction-Engine-Design-Using-DirectX/dp/1590590813/ref=sr_1_1?s=books&ie=UTF8&qid=1322679234&sr=1-1">Introduction to 3D Game Engine Design Using DirectX 9 and C#</a> - This was the book that helped to somehow "flip the switch" and help DirectX make sense after reading and coding only a couple chapters... (I skimmed through 3-4 of the others and never read the rest)<br />
<a href="http://www.two-kings.de/">http://www.two-kings.de/</a> - Some help with the clear explanation/tutorials<br />
<a href="http://zophusx.byethost11.com/tutorial.php?lan=dx9&num=0">http://zophusx.byethost11.com/tutorial.php?lan=dx9&num=0</a> - HUGE help. This guy does through DETAIL so that I could understand. Heavily considered switching to C++....<br />
<a href="http://www.toymaker.info/Games/html/lighting.html">http://www.toymaker.info/Games/html/lighting.html</a> - Helped my understanding of shaders.<br />
<a href="http://www.rastertek.com/tutindex.html">http://www.rastertek.com/tutindex.html</a> - Helped a LOT in understanding and writing shaders for multi-texturing and special effects.<br />
<br />
And of course <a href="http://www.wikipedia.org/">Wikipedia</a>... (<a href="https://wikimediafoundation.org/w/index.php?title=L11_1129_SH_wond/en/US&utm_source=B11_1130_Susan&utm_medium=sitenotice&utm_campaign=C11_1140_SH_GSvWond_US&language=en&uselang=en&country=US&referrer=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FMain_Page">if you haven't donated, but you use it, you should!</a>) and the DirectX Documentation. It's MUCH easier to read it now that I've grasped the major concepts though I undoubtedly have quite a bit to go. And finally thank God for the Internet and search engines... if you're persistent you'll find what you need.<br />
<br />
<span class="Apple-style-span" style="font-size: large;">The JDX Engine</span><br />
So that's my newest passion- building this managed 3D engine- I couldn't really think of a name and finally settled with "JDX". When I've made a fair amount of progress, I actually want to build a <a href="http://en.wikipedia.org/wiki/Recoil_(video_game)">Recoil </a>clone (I played this game when I was 13 and I have<a href="http://www.youtube.com/watch?v=e9CBPfJbqW4"> fond memories</a> of it), since that would be easy to do and not require much artistic skills. If it turns out well, I'd love to also make this 3D engine public for anyone who wants to be able to write a managed DirectX 3D game without being a DirectX expert. It will probably be fashioned somewhat after Blitz3D API, since I've always found it to be extremely intuitive.<br />
<br />
I'll probably give updates on JDX here and there when I can. Mostly I'm teaching myself, so I'm making a lot of mistakes. I'm very open to learning how to actually write an efficient and good D3D code base since a lot of examples *work* but there seems to be many different ways to do things in D3D and I want to not just perform the task but do so efficiently. <a href="http://jhottengineering.blogspot.com/2011/11/pet-peeve-examples-with-warnings.html">Which brings me to one of my new major pet peeves: in most tutorials there seems to be very little out there as to how to actually <i>best write</i> the code.</a> (E.g. do you store a vertex buffer for each object in your world or do you attempt storing them all in the same buffer? What about object parts? How do you apply the correct shaders when you do so? Ect...)<br />
<br />
Until next time....<br />
<br />
<i>UPDATE: Bought another couple books from amazon on 3D Mathematics and DirectX 10; it's helping lots. Engine is coming along well... here <a href="http://jhottengineering.blogspot.com/2011/12/jdx-update-mqo-support.html">are</a> <a href="http://jhottengineering.blogspot.com/2011/12/limited-giles-support-in-jdx-engine.html">some</a> <a href="http://jhottengineering.blogspot.com/2011/12/what-cant-be-measured-cant-be-improved.html">updates</a>.</i>The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com2tag:blogger.com,1999:blog-7176409962679191196.post-13609366097499707092011-11-30T09:34:00.000-08:002015-08-23T16:18:35.991-07:00NodeJS & Simple C# HTTP Server<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
So... I've read quite a bit about <a href="http://nodejs.org/">NodeJS </a>and was very early introduced to it by one of my good friends, Dominic ( <a href="http://dominicbarnes.us/">http://dominicbarnes.us/</a> ). He's a huge JavaScript fan and so NodeJS was a big hit. I like Javascript, but admittedly I am nowhere near as proficient as he is nor do I understand fully the functional paradigm needed to use it to its full extent. (He's great at pretty much everything open source/other side of the MS/Windows fence, so if you need someone like that go hire him and pay him lots of money - you won't regret it.) NodeJS got attention with it's claims to efficiency with its non-blocking programming style- the functional JavaScript would in theory make writing such code easy.<br />
<br />
<span style="font-size: large;">The Bleeding-edge Event Model in NodeJS?</span><br />
This was a couples months ago and I was fascinated by the potential NodeJS performance gains and decided I would try writing a simple server in C# with the same model. I figured since C# is compiled, it may be at the least somewhat quicker than the Windows version of NodeJS. Of course, now it all seems silly having learned what I did about IIS through the process. I never realized most of the comparisons for performance were against Apache, and IIS already performs better than Apache anyway. Of course, that seems obvious now- what was I expecting? I suppose at times we can all be suckers for the success stories of the underdog coming out on top, but in practice that is generally not the case. Regardless, it was a fun learning experience. My proof-of-concept server did turn out to be incredibly fast, (and considerably faster than the Windows NodeJS at the time) but that doesn't mean much considering it offered limited functionality. It was a simple test; I used .NET's HTTP Request classes and didn't build my own implementation to keep simple. (which would have been a huge part of the effort) It was really fun trying to think of ways to optimize my little server. (request handling/caching/reading from disk/ect.)<br />
<br />
<span style="font-size: large;">A C# Server Like NodeJS</span>
<br />
Basically it is a C# application that only has a task bar tray icon for an interface (I never actually got far enough to turn it into a service and separate the UI from the server service) and sits around waiting for requests. A main thread is the one that just sits around listening to the HTTP port[s] all the time and whenever a requests comes through hands it off to a worker thread. The worker thread checks if the request is cached in memory and if so returns it, if not depending on the extension it will either return a static file, image, or compile the script page, add it to the cache, and return it. (or just run the script page's compiled code if cached)<br />
<br />
Having one thread completely devoted to listening for requests and passing the actual request handling to a thread pool allowed my server to respond to requests extremely quickly- this was the concurrency I was after. The cached compiled scripts would run quickly once the compilation for the page script was cached, as it was literally like running a function that returned a string. (Plus obviously the JIT will also compile to native code once the function is called for the first time)<br />
<br />
Yes, it compiles on demand! But that wasn't as big a deal as I thought it would be. It was much easier than I thought since .NET comes with compiler libraries for C#. All my server does is some string parsing on the requested file to look for '@{' and '}@' symbols to know where the C# code begins and ends. (C# "script", anyone?) As I mentioned, compiled methods are kept in memory so that subsequent requests are extremely fast.<br />
<br />
I realize NodeJS operates a bit differently. NodeJS listens on a single thread (main event loop) and when a request comes in, it immediately processes it. For requests what would require "blocking" functions, (such as File I/O... though technically <i>any </i>function call is blocking by definition), a callback is given and the "blocking" function is queued in a thread pool- this way the main even loop thread goes right back to processing and listening to requests. When the "blocking" function completes, it calls the callback function on the main event loop and the request is finished on that main thread. (As I thought through this I began to realize some potential shortcomings)<br />
<br />
In my case, I simply created the event loop to<i> only </i>listen for requests, and then hand off request handling to the thread pool. This is because I couldn't be guaranteed that the page script wouldn't perform a "blocking" operation. However, had I continued with the project and had my intention been to imitate NodeJS exactly, I would have probably needed to build a library of functions that page scripts could have called to handle "blocking" functions. This is where the power or ease of the functional programming style of JavaScript would have been nice. Doing this in C# would have been ugly, but it would be much easier to adopt Javascript's call-back functional style to help segregate those "blocking operations" that should be executed in the thread pool from those that should run in the main event loop. (Technically I didn't have a "main event loop" since all mine did was handle and hand off the requests, but you get the point) In the end, it didn't really matter that I didn't go through all the lengths to simulate that, because for my tests I wrote a page script that didn't do any file I/O or any other "blocking" operations. (Which I suppose makes for poor tests, but good enough for what I needed)<br />
<br />
<span style="font-size: large;">Realizations and Some Final Thoughts on NodeJS & IIS</span><br />
Throughout the whole time I was researching more about the IIS pipeline and began to realize IIS does pretty much the same thing as my server did as far as listening, handling off requests, and working with a thread pool to process them. (of course, it does it a whole lot better) Eventually I stopped development; <a href="https://github.com/JhottMaster/EventModelServer">the code is here if curious</a>. (Since I abandoned the project, the "scripting" support is very limited- it supports C# since it uses the .NET compiler but the page script code doesn't have access to any server variables like POST/GET, ect making it close to useless)<br />
<br />
So NodeJS/IIS thoughts. I think it is a cool technology and I've still got a lot to learn on the subject, but I've researched enough where I feel I have a fair opinion. I think IIS does a pretty darn good job and NodeJS model isn't exactly ground-breaking here... it's actually been around for a long time. Apache is the big web server it always seems to be compared to, and I suppose that's where there's a big win performance-wise is since Apache spawns a new thread for every request. (Maybe the just need to implement a thread pool in their pipeline?) My thoughts are you'd be hard pressed to get test results (and not just mass concurrent request tests) where an equivalent MVC.NET page written well and using IIS under-performs it's equivalent NodeJS page.<br />
<br />
In closing, here are a couple articles I agree with, though I think he is rather harsh/offensive to the NodeJS community, but he seems to hit the nail on the head as far as analyzing performance and the NodeJS model:<br />
<a href="http://teddziuba.com/2011/10/node-js-is-cancer.html">http://teddziuba.com/2011/10/node-js-is-cancer.html</a>
<br />
<a href="http://teddziuba.com/2011/10/straight-talk-on-event-loops.html">http://teddziuba.com/2011/10/straight-talk-on-event-loops.html</a>The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-10445299423939535652011-11-30T09:08:00.000-08:002015-08-23T16:18:45.216-07:00Part 2: SqlBulkCopy Class (MS SqlServer and .NET)<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
<i style="background-color: white; color: #333333; font-family: Georgia, serif; font-size: small; line-height: 20px; text-align: left;">At work we're making an effort to contribute technical knowledge to a centralized IT wiki. I like that. Writing encouraged a good understanding of the technology and it's benefits. I didn't think I'd have much to write but I'm surprised how some things I take for granted as simple, others didn't know and vice versa. This two part series is from those entries.</i><br />
<i style="background-color: white; color: #333333; font-family: Georgia, serif; font-size: small; line-height: 20px; text-align: left;"><br />
</i><br />
<br />
<div class="ms-rteFontSize-2" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 10pt; line-height: 18px; margin-bottom: 1em;">
<strong>Introduction</strong></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; line-height: 18px; margin-bottom: 1em;">
</div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; line-height: 18px; margin-bottom: 1em;">
Normally, inserting rows into SQL server is quick and easy and done through a simple INSERT SQL statement. This is fine when saving data to one, two, or even a few rows. However, when it is necessary to insert larger sets of data, this method becomes not only functionally inadequate, but slow and clunky. In this entry (part two of a two-part series) I wanted to write about the second option we will look at for inserting large sets of data: using .NET’s SQLBulkCopy class.<b><span style="font-size: 12pt;"></span></b></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; line-height: 18px; margin-bottom: 1em;">
<b><span style="font-size: 12pt;"><span class="ms-rteFontSize-2" style="font-size: 10pt;">The SqlBulkCopy Class</span></span></b>When it is necessary to insert more than about a 1000 rows of data, a <a href="http://nfxtulsp2010/DACS/Blog/Lists/Posts/Post.aspx?ID=37" style="color: #0072bc; text-decoration: none;">TVP</a> now begins to reach the limits of its performance gain. If we are using the TVP’s only for inserts, we can move up and dramatically increase performance by using .NET’s SqlBulkCopy class. In addition to providing the functionally for large inserts, the SQLBulkCopy class can also be used to copy large amounts of data between tables. With SQLBulkCopy we can deal with <i>millions</i> of rows if need be. Here is an amazing whitepaper on the SqlBulkCopy class’ performance:<a href="http://www.sqlbi.com/LinkClick.aspx?fileticket=svahq1Mpp9A%3d&tabid=169&mid=375" style="color: #0072bc; text-decoration: none;">http://www.sqlbi.com/LinkClick.aspx?fileticket=svahq1Mpp9A%3d&tabid=169&mid=375</a></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; line-height: 18px; margin-bottom: 1em;">
<b class="ms-rteFontSize-2" style="font-size: 10pt;"><span style="line-height: 14px;">The SqlBulkCopy Class</span></b><br />
Using the SQLBulkCopy class if fairly simple. The dataset you use must match the columns on the table. If the order or column names are a bit different, that’s okay since that can be handled with the SqlBulkCopy class’ ColumnMapping property which is just for that. Here’s a .NET sample of using the SqlBulkCopy class to update a table named “tblSIWellList” from a table name “MyData” within a DataSet:</div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="color: blue; font-family: consolas; font-size: 9.5pt;">Using</span><span style="font-family: consolas; font-size: 9.5pt;"> objConnection <span style="color: blue;">As</span> System.Data.SqlClient.SqlConnection = GetSQLConnection()</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> objConnection.Open()</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> </span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> <span style="color: blue;">Using</span> bulkCopy <span style="color: blue;">As</span> SqlBulkCopy = <span style="color: blue;">New</span> SqlBulkCopy(objConnection)</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> bulkCopy.DestinationTableName = "dbo.tblSIWellList"</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> <span style="color: blue;">With</span> bulkCopy.ColumnMappings</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> .Add(<span style="color: blue;">New</span> SqlBulkCopyColumnMapping("ID", "ID"))</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> .Add(<span style="color: blue;">New</span> SqlBulkCopyColumnMapping("EPD", "EPDate"))</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> .Add(<span style="color: blue;">New</span> SqlBulkCopyColumnMapping("Comments", "Comments"))</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> .Add(<span style="color: blue;">New</span> SqlBulkCopyColumnMapping("Date", "Date"))</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> .Add(<span style="color: blue;">New</span> SqlBulkCopyColumnMapping("RTP_Date", "RTPDate"))</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> <span style="color: blue;">End</span> <span style="color: blue;">With</span></span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> bulkCopy.WriteToServer(ds.Tables("MyData"))</span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="font-family: consolas; font-size: 9.5pt;"> <span style="color: blue;">End</span> <span style="color: blue;">Using</span></span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; margin-bottom: 0.0001pt;">
<span style="color: blue; font-family: consolas; font-size: 9.5pt;">End</span><span style="font-family: consolas; font-size: 9.5pt;"> <span style="color: blue;">Using</span></span></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; line-height: 18px; margin-bottom: 1em;">
<br /></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; line-height: 18px; margin-bottom: 1em;">
Because SqlBulkCopy class is designed to copy/insert a large number of rows, transactions are handled in batches. It is possible to specify how large each batch is (e.g. 5000 rows at a time) but by default a single transaction (“batch”) is used for all rows. When committing transaction in batches, a failed batch will only roll back the last active transaction in the batch. (This may not necessarily be <i>all</i> rows if a previous batch was successfully committed)</div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; line-height: 18px; margin-bottom: 1em;">
<br /></div>
<div class="MsoNormal ms-rteFontSize-2" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 10pt; line-height: 18px; margin-bottom: 1em;">
<b>Considerations when using .NET’s SlqBulkCopy class</b></div>
<div class="MsoNormal" style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; line-height: 18px; margin-bottom: 1em;">
There are a few “gotcha”’s to keep in mind when using the SqlBulkCopy class:</div>
<ul style="background-color: white; color: #4c4c4c; font-family: Verdana, Arial, sans-serif; font-size: 11px; line-height: 18px; margin-top: 0in;" type="disc">
<li class="MsoNormal">When the source and destination table data types are different, SqlBulkCopy will attempt to convert to the destination data type where possible but this will incur a performance hit.</li>
<li class="MsoNormal">By default, PK’s are assigned by destination and are not preserved.</li>
<li class="MsoNormal">By default, constraints are not checked and triggers are not fired. Also row-level locks are used. Changing these setting may affect performance.</li>
</ul>
The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-92007529002650370742011-07-14T11:10:00.000-07:002015-08-23T16:18:52.892-07:00Part 1: Table Valued Parameters (SQL Server 2008+)<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
<span style="font-size: small;"><i>At work we're making an effort to contribute technical knowledge to a centralized IT wiki. I like that. Writing encouraged a good understanding of the technology and it's benefits. I didn't think I'd have much to write but I'm surprised how some things I take for granted as simple, others didn't know and vice versa. This two part series is from those entries.</i></span><br />
<span style="font-size: large;"><strong><br />
</strong></span><br />
<span style="font-size: large;"><strong>Introduction</strong></span><br />
Normally, inserting rows into SQL server is quick and easy and done through a simple INSERT SQL statement. This is fine when saving data to one, two, or even a few rows. However, when it is necessary to insert larger sets of data, an array, or some kind of dynamic list atomically, this method becomes not only functionally inadequate, but slow and clunky. This may also be the case when updating a large number of records individually. In this entry (part one of a two-part series) I wanted to write about the first of two options we will look at for inserting/updating larger sets of data: Table Valued Parameters.<br />
<br />
<div>
</div>
<span style="font-size: large;"><strong>Using Table Valued Parameters in SQL Server 2008</strong></span><br />
When it is necessary to insert or update more than just a few rows or a list of data, a TVP (table valued parameter) is the next step. TVP’s became available in SQL Server 2008 and are perfect for sending a stored procedure a dataset of anywhere from a few rows to around a thousand. <br />
<br />
<div>
</div>
To demonstrate, I am going to use some samples from a recent project that I updated to use TVPs. In order to create a stored procedure that can take a table as a parameter, first we create a UDT (User Defined Type) of the table type. You create your user defined table type (UDTT) just like you would normally create a SQL server table:<br />
<div>
</div>
CREATE TYPE [dbo].[typProjectStreamLight] AS TABLE(<br />
[PCID] [int] NOT NULL,<br />
[Stream] [varchar](100) NOT NULL,<br />
[StreamType] [varchar](20) NOT NULL,<br />
[BudgetMonthLabel] [varchar](20) NOT NULL,<br />
[Value] [decimal](18, 6) NULL<br />
) <br />
<div>
GO</div>
<br />
<div>
Once you have created the UDTT, you can create a stored procedure that will take the UDT as a parameter:</div>
<div>
</div>
-- =============================================<br />
CREATE PROCEDURE [dbo].[spProjectStreams_AllMonths_set] <br />
@CID INT,<br />
@UserName VARCHAR(25),<br />
@tvpMonthValueList typProjectStreamLight READONLY<br />
AS<br />
BEGIN<br />
<br />
<div>
In this example, the parameter is named “@tvpMonthValueList” and you can see the UDT as the declared data type next to the parameter name. (“typProjectStreamLight”) Note that to use UDTT’s as arguments you must declare them as read-only.</div>
<br />
Here’s another sample using TVP’s as a way to emulate a list of data or an array. Here’s the UDTT declaration:<br />
<br />
CREATE TYPE [dbo].[typIntList] AS TABLE(<br />
[Value] [int] NOT NULL<br />
)<br />
GO<br />
<br />
And here the procedure using it as an argument:<br />
<br />
CREATE PROCEDURE [dbo].[spUpdateCalculatedStreamsFromList]<br />
@PCIDList [typIntList] READONLY,<br />
@LastUpdatedBy VARCHAR(25)<br />
AS<br />
BEGIN<br />
<br />
<br />
<span style="font-size: large;"><strong>Communicating between .NET and SQL Server 2008 using TVPs</strong></span><br />
Using a stored procedure with .NET is very simple- the main difference is a SQL Data Type of “Structured” is assigned to the parameter containing the table. Because the System.Data.SqlClient supports populating table-valued parameters from DataTable, DbDataReader or IEnumerable<sqldatarecord> objects, populating the UDTT is very easy. Below is an example with the generic setup for the SQL command same as always plus our new table:<br />
<br />
Dim objConnection As SqlConnection = DataAccess.modConnection.GetSQLConnection<br />
objConnection.Open()<br />
Using objCommand As SqlCommand = objConnection.CreateCommand()<br />
With objCommand<br />
.CommandType = CommandType.StoredProcedure<br />
.CommandText = "spProjectStreams_AllMonths_set"<br />
.Parameters.AddWithValue("@CID", CID)<br />
.Parameters.AddWithValue("@UserName", DBNullIF(UserName))<br />
.Parameters.AddWithValue("@tvpMonthValueList", mrshlTable)<br />
.Parameters("@tvpMonthValueList").SqlDbType = SqlDbType.Structured.ExecuteNonQuery()<br />
End With<br />
End Using<br />
<br />
</sqldatarecord><br />
<div>
In the sample above, “mrshTable” is a DataTable object that matched the schema of the “typProjectStreamLight” UDTT described previously.</div>
<br />
<br />
<span style="font-size: large;"><strong>Considerations when using TVP’s</strong></span><br />
There are a couple of things to keep in mind when using UDTT’s: (At least as of July 2011)<br />
<ul>
<li>As noted above, UDTTs must be declared as read-only when used as an argument in a stored procedure- this means you won’t be able to modify the data in the UDTT.</li>
<li>UDTT’s cannot be passed to a SQL function.</li>
<li>UDTT’s cannot be modified after created- they must be dropped and created again. </li>
<li>UDTT’s cannot be dropped if there are any stored procedures currently using them as arguments. First the stored procedure must be dropped or altered to not use the UDTT and then the UDTT may be dropped.</li>
</ul>
Here are some benefits you may reap by using TVP’s correctly:<br />
<ul>
<li>Update/Insert operations can be performed with less network overhead.</li>
<li>Using a UDTT in a stored procedure allows performing your INSERT/UPDATE operations in a single set allowing for a faster, more efficient SQL Server performance</li>
<li>By using a stored procedure that accepts a UDTT, using one connection, running the whole operation in a transaction, and staying away from ad-hoc SQL become much easier and the default.</li>
<li>I’ve found it may be much easier to write/refactor code in .NET to use a table and simply pass it to the stored procedure when we are ready to insert/update the data.</li>
</ul>
<br />
<div>
</div>
<br />
<div>
</div>
<br />
<div>
</div>
The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-33049247610407260412010-09-20T09:50:00.000-07:002015-08-23T16:19:01.329-07:00Be careful: static classes can persist across HTTP Contexts!<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
This morning I was doing some quick testing of some functionality I had added to our web application- (still in development)- when I realized something with security was wrong. It seems like the roles were messed up. I'd log in with one user and I was missing functionality. I tried logging back as another user and then everything was fine. I restarted the application and this time it worked with the other user but not the one I first began testing with. I could not figure out what was wrong- I hadn't touched security at all recently!<br />
<br />
Well, today is Monday. Friday evening, right after I checked in my changes, a coworker checked in his. He had quickly added functionality to cache the role items that are used to determine a user's right. I was glad we had finally gotten to this, because this was an expensive call to the database. I didn't think much of it, it never occurred to me it had anything to do with my problem.<br />
<br />
I started digging in slowly until finally when I logged in as a different user I noticed the roles were not refreshed. Looking into it deeper it started dawning on me that this may be because our Security class was static- and we were caching the role items in a private static variable. Surely this wasn't the case- I mean, every page call is like a new execution context, it's own HTTP Context, etc... yet it made total sense- that IS how C# operates.<br />
<br />
Immediately I called my coworker over and we talked about it and put a proof of concept together. It was simple- a form with a text box and a label - it would call a static class that had a private static property, this property would be used to populate the label and we added functionality to update this static class' property from a button on the page.<br />
<br />
Sure enough- even in two different browsers (Chrome and IE) if we were looking at the same page we could update the label on one browser from the other.<br />
<br />
It makes total sense - any different case would imply a new "instance" of this static class is being created per HTTP Context or Session, which would totally violate the C# spec.<br />
<br />
The .NET model does such a good job at abstracting away from the reality that a page is not within it's own "execution context" (so to speak). So the code on your page is not 100% segregated... it's so obvious, so simple, but it's easy to slip into thinking that it is, since everything <span style="font-style: italic;">seems </span>to work that way.<br />
<br />
Think about it- every time a page is called, we "get new objects" for what seems to be everything- new page, new controls, new context, etc. It's easy to forget ASP.NET is still just instantiating an HTTP Context and our Page class and passing it to the thread pool, but it is still all within the same application domain. Reminds me of one of my favorite articles from Joel, <a href="http://www.joelonsoftware.com/articles/LeakyAbstractions.html">The Law of Leaky Abstractions</a>.<br />
<br />
Anyway... we fixed the caching- it's saved per session.But remember - it's important to understand how the ASP.NET worker process operates, and that all our classes and objects are still being instantiated within the same application domain.The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-59205577589576385222010-09-16T10:51:00.000-07:002015-08-23T16:19:06.078-07:00Linking to a JavaScript file...<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
So we have these big JavaScript functions in one of our MasterPages. I wanted to move them to a separate file but it proved more difficult than I thought!<br />
<br />
I could not add a "<link>" or "<script>" tag to the get and use an application relative path such as "~/Scripts/MyScript.js".<br />
<br />
Ended up just creating the control dynamically but it was interesting how I had to arrive to the right path. Here it is:<br />
<pre class="csharpcode"><span class="rem">// Retrieve Header</span>
HtmlHead header = (<span class="kwrd">this</span>.Page.Header <span class="kwrd">as</span> HtmlHead);
<span class="rem">// If we have access of the page header:</span>
<span class="kwrd">if</span> (header != <span class="kwrd">null</span>)
{
<span class="rem">// Add script block for Master functions:</span>
HtmlGenericControl scrptBlck = <span class="kwrd">new</span> HtmlGenericControl(<span class="str">"script"</span>);
scrptBlck.Attributes.Add(<span class="str">"src"</span>,
VirtualPathUtility.MakeRelative(
Request.AppRelativeCurrentExecutionFilePath,
<span class="str">"~/Scripts/MasterFunctions.js"</span>));
scrptBlck.Attributes.Add(<span class="str">"type"</span>, <span class="str">"text/javascript"</span>);
header.Controls.Add(scrptBlck);
}</pre>
The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-74056228392144984282009-02-21T13:16:00.001-08:002015-08-23T16:19:12.052-07:00IE POST and redirect errors...<b><i>This blog post only exists to prevent expired linking. My blog has moved to: <a href="http://pabloaizpiri.com/">http://pabloaizpiri.com/</a></i></b><br />
<br />
--------------------------------------------------------------------------------<br />
<br />
<i><span class="Apple-style-span" style="font-family: inherit; font-size: x-small;"><b>UPDATE NOTE</b>: I did find a much simple solution to this; as mentioned in the comments at the bottom, <span class="Apple-style-span" style="background-color: white; line-height: 20px;"> I found that sending a compressed P3P security header did just fine to address the redirect issue and was much more simple to implement.</span></span></i><br />
<i><span class="Apple-style-span" style="font-family: inherit; font-size: x-small;"><span class="Apple-style-span" style="background-color: white; line-height: 20px; text-align: left;">The HTTP module is now modified to send the P3P header on redirect. You can do so as follows:</span></span></i><br />
<div>
<i><span class="Apple-style-span" style="font-family: inherit; font-size: x-small;"><span style="background-color: white; line-height: 20px; text-align: left;">_context.Response.AddHeader("p3p", "[P3P Policy Here]")</span></span></i></div>
<div>
<i><span class="Apple-style-span" style="font-family: inherit; font-size: x-small;"><span class="Apple-style-span" style="background-color: white; line-height: 20px; text-align: left;">You can find more information about P3P </span><a href="http://en.wikipedia.org/wiki/P3P" rel="nofollow" style="background-color: white; line-height: 20px; text-align: left; text-decoration: none;">here</a><span class="Apple-style-span" style="background-color: white; line-height: 20px; text-align: left;">.</span></span></i><br />
<br />
Many developers use redirection as a tool to present the user with another page once a process has been completed. One of the most common uses is- for example- a login. A user logs in, and- if he is authenticated he is redirected to another page, if not the same page is displayed with an error message informing the user why the authentication failed. (Such as an incorrect username/password combination)<br />
<br />
I don't know if I would consider this the best way to carry out these types of transactions, since in my understanding that isn't exactly what redirects were meant for- however, it is a simple way to address these type of situations and on strictly pragmatic terms- can often be the best approach.<br />
<br />
I've been working on a page that does exactly this, when I ran into an error.<br />
<br />
<br />
<span style="color: #5688aa; font-size: 16pt;">The Problem.</span><br />
<br />
Apparently, if you do a POST to a page and that page sends back headers to set a cookie <span style="font-style: italic;">AND</span> redirect in the same response, the cookie does not get set. This seems to be a bug in IE. And the fact that the page was originally called from an iframe <span style="font-style: italic;">seemed</span> to be key to the problem. <br />
<br />
How did I run into this? I was working with another system that seamlessly integrated into the web application, and it happened to open the website in an iframe. The process was mostly simple:<br />
<ol>
<li>Client used Web Application A which makes a form post to Web Application B within an iframe. (Web Application A uses iframes)</li>
<br />
<li>Web Application B processes the post for authentication.</li>
<br />
<li>If the authentication succeeds, Web Application B redirects the client to a "services" pages also on Web Application B that offer services.</li>
<br />
<li>The client is now looking at a "services" page on Web Application B which is within an iframe of the Web Application A.</li>
</ol>
The problem was between steps 3 and 4 - in that the 4th step would never happen. The client was certainly redirected to the "services" page but the "services" page would redirect the client right back to a login page because the client was not authenticated. In step 3 the server would attempt to send back a cookie to identify the session and redirect in the same response, but the cookie would never be set and never sent back.<br />
<br />
It was an "intermittent" problem- and I still haven't gotten it to replicate more than twice to figure our the exact conditions under which it <span style="font-style: italic;">does</span> work; the rest of the time it failed. <br />
<br />
However, a suspicion tells me it was not an intermittent error, but rather that it <i>always</i> happens. (After all, there are no intermittent errors. It's all a matter of finding out the exact conditions under which these anomalies occur)<br />
I noticed that when the browser <i>did</i> successfully set the cookie and return it, it was most likely because I had previously signed on the site before, effectively forcing the browser to send back my old authentication cookie- "seemingly" making it work.<br />
<br />
Anyway, I found a blog by Brian Donahue where it seems like he ran into the same bug, or at least a related bug:<br />
<a href="http://www.persistall.com/archive/2008/01/25/cookies--redirects--nightmares.aspx" target="_blank">http://www.persistall.com/archive/2008/01/25/cookies--redirects--nightmares.aspx</a><br />
<br />
It was the closest thing I could find to the problem I was having. After attempting the solutions provided with no success, it seems to me like there is a little more to it, and in this case, the only thing I can think different is the fact that in my case it involves an iframe. However, something tells me that this can't be it. For one, it doesn't make sense, but I'm also not completely ruling it out- there just may be something about the way cookies are handled within an iframe that I don't know about, but basically I'm still wondering why the solutions Brian Donahue wrote about did nothing for me. I would be more than appreciative to anyone who could educate me better on why this happens- for now I will continue to attempt to find the exact reason for this bug! <br />
<br />
<br />
<span style="color: #5688aa; font-size: 16pt;">The Consequences.</span><br />
<br />
Anyway, this cookie not being set was a rather large problem, because this would mean the user would not be authenticated. This would destroy the communication between both systems so I had to think of some way to fix the problem.<br />
<br />
I started looking into the session object in ASP.NET and how that is handled, as well as just all the HTTP Pipeline in general.<br />
<br />
I thought about enabling cookie-less sessions, and it might have worked out if I could use them on a per-page basis, but that was quite a port-over and not as secure. I hated the idea of an ID being displayed on the address bar, (especially for each pages) both for security and just aesthetic reasons, really. All in all, I finally decided against enabling cookie-less sessions.<br />
<br />
<br />
<span style="color: #5688aa; font-size: 16pt;">Addressing The Problem.</span><br />
<br />
How was this problem solved?<br />
<br />
I wrote an HTTP Module. The module essentially "interrupts" the response, checks if the response is a redirect and sets a cookie- and if it does- it changes the URL direction by appending to the url a session ID, (like the cookie-less session feature) and a key that quickly expires. <br />
<br />
The receiving page "interrupts" the request, checks for this session ID, generates a cookie with the ID value from the query string, and feeds this artificial cookie into the HTTP the pipeline. The session is then loaded normally by ASP.NET. <br />
There is also some logic to check the key that was passed in for validity, as well to check if it was expired- if so, the session is destroyed effectively logging the user out before any of the code in the page object is executed.<br />
<br />
I felt this solution was great for a few reasons:<br />
<ul>
<li><b>The URL will virtually never display a session ID.</b><br />For 98% for the application usage, the URL will never contain a session ID. When it does, it will be more secure on account of the expiring key. It will only show on those specific URLs that redirect <i>and</i> attempt setting a cookie at the same time.<br />Currently the only page that redirect this way is linked from an iframe, which means the user will never see the page address; now I can lean towards 99.99%</li>
<br />
<li><b>More Secure than the enabling session-less cookies.</b><br />The security is much tighter than cookie less sessions because an expiring key must also be authenticated when the session data is passed on.</li>
<br />
<li><b>It only affects redirects that set cookies.</b><br />This was great in that I didn't want to add complexity to the website and overhead that was unnecessary.</li>
<br />
<li><b>It is completely transparent to other developers.</b></li>
</ul>
<b>The last point</b> was one of the greatest benefits. This means any other developer can come after me, and write his pages like he normally does, without ever even being aware of the IE Redirect bug. Because the code works on a lower level in the HTTP Pipeline chain, this is all done transparently.<br />
<br />
This drives the cost of ownership of the application down because future developers will not have to be versed on the changes or develop specifically to accommodate for the change, or even run into the error in the first place and attempt to address it. This drives development time down which translates to more time in other projects and no money wasted. <br />
<br />
Until next time,<br />
- Jheatt</div>
The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com5tag:blogger.com,1999:blog-7176409962679191196.post-71736567503012269242009-01-16T11:47:00.000-08:002009-02-21T14:14:07.464-08:00XML, Web Services, SOAP, and transmission protocols<span style="font-size:16pt; color: rgb(86,136,170);">Intro</span><br /><p>In the pursuit of integration, I've found myself needing to integrate with clients with different integration needs/capabilities. In doing so I found some interesting things about .NET as well as just web services in general.</p><br /><span style="font-size:16pt; color: rgb(86,136,170);">Communication Protocols</span><br /><br />Although most web services are consumed over HTTP as the transmission protocol, not all communicate through SOAP. While generally the standard is communication through SOAP, some communicate over HTTP.<br /><br />I recently found .NET does have the ability to use HTTP POST and GET as communication protocols as well, which seems expected. However, as of .NET 1.1 SOAP is the primary communication protocols and while HTTP and GET <i>are</i> supported, they are disabled by default. It turns out that it is very simple to enable them, only requiring the following change in your application's "web.config" file:<br /><blockquote style="background-color: white; color: black; font-family: "Courier New" Courier monospace; background-color: white;"><pre><webservices><br /> <protocols><br /> <add name="HttpPost"><br /> <add name="HttpGet"><br /> </protocols><br /></webservices></pre></blockquote><br />This makes integration with many legacy systems much more possible- and especially other systems that are not on the .NET platform.<br /><br />A page built to make a request to a web service using HTTP as both the transmission and communication protocol may look as such:<br /><br /><blockquote style="background-color: white; color: black; font-family: "Courier New" Courier monospace;background-color: white;"><pre><form action="mywebservice.asmx/WebServiceMethod" method="post"><br /> <input name="parameterName" value="param1" /><br /> <input name="parameterName2" value="param2" /><br /> <br /> <input type="submit" value="Submit!" /><br /></form></pre></blockquote><br />If you were developing your web service on .NET all you'd have to do is develop as normal and modify your web.config as described. You may also consume a web service using GET and query parameters.<br /><br />One of the scenarios I've found myself in is a client desiring to consume the web service using HTTP as both the transmission and communication protocols, however passing XML as a "parameter" effectively using XML as a communication protocol as well. I found this is relatively simple and easy to work with as long as the XML is url-encoded as one of the form's fields, and is decoded once loaded. (In .NET you would simply use HttpUtility.UrlDecode()).<br /><br />You can then load this string in an XmlDocument and manage it easily using the object structure in .NET. When returning a response- if the client is able to accept an XML response- using .NET you can simply return the XmlDocument object. The XmlDocument Object can be found in the System.Xml namespace. <i>do not return an XML-formatted string if you have an XmlDocuments.</i><br /><br /><br /><span style="font-size:16pt; color: rgb(86,136,170);">XML and SQL Server 2005</span><br /><br />In the midst of all this, I found myself storing XML. I realized there was an XML data type in SQL Server, and so I did a bit of research on it. I decided to use it and I was pretty amazed.<br /><br />By storing data as XML in SQL Server 2005 using the XML data type, you can actually use XQuery to write queries that take advantage of the XML data structure. It's almost like running your main query and s asubquery to filter the XML data. It make it very practical and simple to retrieve and store XML data.<br /><br />A SQL Query using XQuery might look like:<br /><br /><blockquote style="background-color: white; color: black; font-family: "Courier New" Courier monospace;background-color: white;"><span style="color:blue">SELECT</span> OrigXmlRequest.query(<span style="color:red">'/Notes//child::*'</span>) <br /><span style="color:blue">FROM</span> YardiRequest</blockquote>This would return all the XML child nodes and their descendants of 'Notes'.<br /><br />XML Data in SQL Server 2005 is stored in the same was a varchar(MAX) is, and has a max size limit of 2 gigs. (Which if you're hitting anywhere close to that 2 GIG limit when storing an XML file, you should probably store the data normalized in tables)<br /><br />One thing I ran into that I would like to point out when storing the XML data from .NET into SQL was the encoding format used. I kept receiving an encoding exception, until I realized the encoding needed to be changed from UTF-8 to UTF-16 in order to be stored in the database. If you have an XmlDocument object, an easy way to do this would be to just edit the first child and the you will be able to store the data. e.g.:<br /><br /><blockquote style="background-color: white; color: black; font-family: 'Courier New' Courier monospace;background-color: white;">xmlDcRequest.LoadXml(myXmlString);<br /><span style="color:green">// Change encoding to UTF-16 for storage in SQL Server</span><br />xmlDcRequest.FirstChild.InnerText = xmlDcRequest.FirstChild.InnerText.ToLower().Replace(<span style="color:red">"utf-8"</span>, <span style="color:red">"UTF-16"</span>);<br /></blockquote><br />...<br /><blockquote style="background-color: white; color: black; font-family: 'Courier New' Courier monospace;background-color: white;"><span style="color:green">// Adding the XML to a stored procedure parameter to store the XML data:</span><br />sqlCmd.Parameters.Add(<span style="color:red">"@XMLRequest"</span>, SqlDbType.Xml).Value = xmlDcRequest.InnerXml.ToString();</blockquote><br /><br /><span style="font-size:16pt; color: rgb(86,136,170);">Final Notes on Web Service Development</span><br /><br />I think one of the biggest lessons I am learning from putting together web services to service other clients is the cruciality of writing code as bulletproof as possible. Of course, you should always be doing this anyways. Make sure your web service is returning clear error codes and descriptions when errors do occur to make development for the client straightforward. Clear communication and a detailed scope of work are essential. Even though these are simple things, it seems many times these things are missed. <br /><br />- JheattThe *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-77697894821914849892009-01-10T11:27:00.000-08:002009-01-10T11:49:19.767-08:00SQL Server Reporting Services and Headers requiring dataset fields.Any search for "fields in headers SSRS" in Google will come up with many search results with pages addressing workarounds for one of Microsoft SQL Server Reporting Service's biggest weakness: the inability to add fields to the header or footer from a dataset. <br /><br />This limitation makes some sense in that a dataset returns rows of fields and so the problem might lie in identifying which row should be displayed, however one should be able to define exactly what to display- the first or last row or maybe an aggregate of the rows.<br /><br />All that to say, this is yet another workaround I haven't seen elsewhere. It addresses the problem of maintaining data in the header regardless of what page the report is being viewed on. <br /><br />Why another work around? I tend to avoid storing fields in internal parameters because I've had problems with it before, and shared variables are a horrible solution in a production enviroment because of the possibility of concurrently running reports. It seems like these are always the solutions given to this issue.<br /><br />So here goes: the way I've solved it is by making a group <em>within the table displaying my dataset</em>, and then using the footer <em>of the group</em> to store a <strong>list box</strong>, (if you are needing to use the footer, simply extend it to add another footer row right below it). In that list box you can place another table or fields and you can even have this list box pull from another datasource. When done, set the group footer's "Hidden" property to "True" so that it doesn't show on the rendered report, and you're set. You can now reference these fields from the header or footer like so:<br /><br /><blockquote>=First(ReportItems!<strong>NameOfTextboxInListItem</strong>.Value,"<strong>datasourceORgroupName</strong>")</blockquote><br /><br />And since it's an expression, you can do whatever you want. You can also use aggregates and such- just make sure you modify the expression on the field/table pulling directly from the dataset, and not the field in the header/footer that will actually be displaying the data.<br /><br />The only major problem I've run into with this approach (that I am still working on) is that the fields in the header will export as blank if I export the report as a PDF. It will do fine in HTML, or even Excel. (Which is funny since most rendering errors seem to be tied to Excel instead of the other way around).<br /><br />If any solution is found to this issue, or if someone can explain to me the PDF rendering process and why these fields show up as blank, it would be greatly appreciated!<br /><br />Hopefully this post was of some use to those that do not need to export in PDF...<br /><br />- JheattThe *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-75286787273405426512009-01-10T09:56:00.000-08:002009-01-10T11:50:43.297-08:00HTTPS, HTTP, and form posting.So this is really more of a quick blog post for my own benefit of recording a solution. HTTPS and form posting.<br /><br />I haven't really done a whole lot with HTTPS and I'm still sort of new to it. I was trying to do a form post cross-domain between a beta server and my computer and it wouldn't work. It would run fine locally, but not with the BETA server. The funny thing is it ran fine when I tried running it locally on the BETA server or on my development machine, but I couldn't post from my development machine to the BETA server- so it <em>seemed</em> like it was a cross-domain security issue.<br /><br />All the POST fields would come up blank- actually they wouldn't even be set, query parameters would be fine of course.<br /><br />Another fellow developer I work with pointed out I should be posting back with HTTPS. The form was posting to a HTTP address, but once it loaded it showed HTTPS. That was the only problem. I couldn't believe it- so simple! I guess I needed to be humbled.<br /><br />So I still don't understand HTTPS completely, but I will definitnely be researching. My guess is that the POST didn't go through since the HTTPS server was addressed as HTTP in the original form POST and so the POST didn't go through all the SSL encryption action and hence it turned up empty on the server side... any one care to give me the technical explenation? I guess I'll find out more when I research.<br /><br />- JheattThe *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-68747755685062568432009-01-10T09:50:00.001-08:002009-01-10T11:49:41.284-08:00New Job; New ObstaclesSo I switched jobs, and this new place is totally a Microsoft Shop. It's different than what I'm used to, but in a good way. I like how we are working with the latest tech for the most part, including MS Visual Studio 2008- love it.<br /><br />Doing nothing but .NET (Both VB and C#) with the exception of a few third-party controls (Telerik) this will was sort of a new field for me with new obstacles. I'm excited. I was looking forward to working with a .NET development team and loving it so far.<br /><br />So I know I haven't written a tech blog forever but I'm starting to realize the importance of this blog to more than just giving back to the community- more like a way to remember all the problems and solutions I've come accross. Sometimes I'll run into the same problem I've ran in the past but will forget the solution, so it's easy to see how a blog would be a great way to retrieve this information while helping others as well.<br /><br />Anyway, just preparing the way for my first post as a software engineer here at Trak-1. Happy coding...<br /><br />- JheattThe *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-60272359252475635182008-11-14T14:42:00.000-08:002009-01-10T11:50:01.084-08:00Time Logging & Cookie Data EncryptionOne of the projects I'm working on is to log how long a visitor is on our website <span style="font-style:italic;">viewing a video</span> and use that to enable a service. Yeah... Who would of thought? <br /><br />Anyway this post is both about the time logging approach utilized as well as cookie encryption. Let me give you a little background before we dive in.<br /><br /><span style="font-weight:bold;">The Conditions.</span><br />The goal is a little tricky because of the set up. I'm writing it in Classic ASP, (not exactly my favorite language). Also normally one would think of storing time logging data on a SQL server, but I didn't have access to one. Another tricky part about logging the time a visitor is viewing the video is that the video being viewed can be upwards of an hour. <br /><br /><span style="font-weight:bold;">The Challenge.</span><br />Using session to track the time data isn't a full solution here since the session will timeout unless I increase that, (which right now isn't an option). If I am to use sessions as a way to keep track of the time spent on the site, I need to continually refresh/access the page to update time logging data as well as to keep the session alive.<br /><br /><span style="font-weight:bold;">The Chosen Approach.</span><br />The idea was making an AJAX call asynchronously "in the background" every 10 mins while the user is watching the video. This would keep the session alive, (which I'm finding out it does not for some reason), and would allow me to track with a 10 min accuracy if the user was on our video page or not, (hopefully watching the video). The advantage here is if that if the user moves away from the page, the updates stop effectively defining when the user began and finished viewing the video. <br /><br /><span style="font-weight:bold;">What Actually Happened...</span><br />As much as I hate to say it, I have to admit- this system will at least partly depend on JavaScript and I can't think of any other practical way to avoid this. I find comfort in the fact that we can thank the major adoption of .NET-driven applications for JavaScript becoming a necessity such that a developer can count on a fairly large user base supporting it.<br /><br />Anyway, not having access to a database, I chose to store time logging information on a cookie. I decided I would store the session ID and the time when the request was made. Of course, this could lead to easy tampering and so I chose to encrypt it.<br /><br /><span style="font-weight:bold;">Encryption.</span><br />The encrypting algorithm I used was RC4. I found it easy to implement, thanks to 4guysfromrolla.com. After reading up on it a little I also felt it was rather secure- provided a strong key is used- especially for this purpose.<br /><br />Little did I know the headaches I would run into. I would encrypt the data and store it in the cookie- but when I read it back, some of the data would be missing. It was very bizarre. I was trying all these ways to make sure I was reading/writing the cookie data correctly. <br /><br />Before long I realized that it probably had to do with the way cookies are stored and just exactly what is and is not legal character-wise. I tried URL-encoding them. This bloated the data and I still had data missing. <br /><br />I threw the whole URL-encoding idea out the window (for the moment) and tried filtering for illegal characters myself. I did some research, but all I could find was that <>:= were the illegal characters and nothing else. So I wrote some filters for that. Still missing data. Tried URL encoding as well as the filtering. Still missing data.<br /><br />Finally I decided I would just write a function to go through each character in a string of encrypted data, and find any characters outside of a safe US-ASCII code range and escape them through a custom method. <br /><br />As I was finishing up I ran into the <span style="font-style:italic;">Escape()</span> and <span style="font-style:italic;">Unescape()</span> methods in Classic ASP- so I gave them a shot before even trying my custom functions. They worked wonderfully. On the other hand, my idea of grouping time data by session ID and then using that to calculate time spent on each session to figure out a total of time spent on the site didn't work out so well, so I am choosing another method. I might write on that later.<br /><br /><span style="font-weight:bold;">Lessons Learned:</span> <br />When writing binary[-like] data to cookies, stick to a known safe subset of US-ASCII characters. I still don't know exactly which character it was that caused the problem, but I'm thinking maybe some odd escape character or something along those lines.<br /><br />At the time of this post I'm still in the process of writing the application, so I'm open to any ideas or suggestions on how anyone else would do it.<br /><br />-JheattThe *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0tag:blogger.com,1999:blog-7176409962679191196.post-46142490429752276132008-11-14T08:31:00.001-08:002009-01-10T11:50:24.499-08:00Jhott Engineering<h2><span style="FONT-WEIGHT: bold">Introduction to Jhott Engineering:</span></h2>So I was thinking today after-yet again - finding another solution to a problem that took me a long time to solve... that I should really consider writing a technical blog.<br /><br />Not that I am the all-knowing software designer- far from that. In many ways, I consider myself a new programmer. However, I find that many of the problems I run into many others do as well and find themselves stuck.<br /><br />Realizing much of the time the credit for being able to solve problems goes to other blogs I've read, or just a whole lot of hunting, I decided I should give back and write as well.<br /><br /><br /><h3><span style="FONT-WEIGHT: bold">Who am I?</span></h3>To introduce myself, I'll start with a little history:<br /><br /><span style="FONT-WEIGHT: bold">1996-1998:</span><br />I was intrigued with computers every since I was a young boy. I was introduced into my "computer career" when I encountered an error on my dad's computer when trying to play a video game. I was 9 years old, and this was back in the Windows 95-98 days. I decided I would start browsing through the files to fix it, so I could continue playing my game. Needless to say, I never fixed the game, but it was the beginning of my interest.<br /><br /><span style="FONT-WEIGHT: bold">1999-2000</span>:<br />By the time I was 11-12 years of age, I got my long awaited RIS. Robotics Invention Set. I had always loved Legos, but this Legominstorms set was amazing- I could build robots! And so.. I dived into that. The set came with a visual programming software, but one day when I got an expansion set and read up on challenges, I noticed they always had the "harder" challenges for those who had the "SDK". Of course, I didn't even know what an SDK was that the time, but I was determined to find out.<br />Before long I ran into NQC and started programming using this C-Style language. It was great.<br /><br /><span style="FONT-WEIGHT: bold">2001-2005:</span><br />So now I wanted to write games! So badly! I found "Game Maker" and built a few games. That kept me going for quite a while, but it was only 2D and I wanted to write 3D Games. I played with the demo version of 3DRAD [for those of you who know what that is], and eventually BLITZ3D. Using BLITZ3D and a VB-style language I learned more and more until I was writing networking applications as well, chats, file transfers, ect.<br /><br /><span style="FONT-WEIGHT: bold">2006-2008:</span><br />I went to Teen Mania's Honor Academy. I wanted to work in their IT department so bad! And I did... I started writing websites. Learned HTML ina couple days, a good amount of PHP in a week, and eventually the rest came.. SQL, ColdFusion [shudder], JavaScript, etc. I also did some BI using MS SQL Reporting Services.<br />During my second year I wrote quite a few apps, and I was doing contract work as well. I eventually started delving into .NET as well, quickly falling in love with C#.<br /><br /><span style="FONT-WEIGHT: bold">Present:<br /></span>I currently am working for good-size ministry- I started not too long ago. There's quite a bit of opportunities to improve the systems, so that's always exciting.<br />I also do contract work for a couple other parties, as well as still enjoy personal programming projects and robotics. (Trying to fix that old RIS set!)<br /><br /><h3><span style="FONT-WEIGHT: bold">Conclusion:</span></h3>I'd love to hear any comments for improvements, suggestions, etc- I love feedback. I'm here to learn and I've got a long ways to go. I'm still young and I've got my life ahead of me. I also cannot write anything about myself without mentioning that Jesus Christ is my Lord and Savior and has been my driving force in my life since I was 16 years old. <br /><br />- Jheatt.The *NEW* Pablohttp://www.blogger.com/profile/04348743696627086264noreply@blogger.com0