<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2026-05-13T03:52:16+00:00</updated><id>/feed.xml</id><title type="html">Sarp’s Blog</title><subtitle>A blog about writing software for groups playing games together.</subtitle><author><name>SarpedonTD</name></author><entry><title type="html">Designing IT for my EVE Echoes Alliance</title><link href="/2024/09/14/Designing-IT-For-Echoes.html" rel="alternate" type="text/html" title="Designing IT for my EVE Echoes Alliance" /><published>2024-09-14T00:00:00+00:00</published><updated>2024-09-14T00:00:00+00:00</updated><id>/2024/09/14/Designing-IT-For-Echoes</id><content type="html" xml:base="/2024/09/14/Designing-IT-For-Echoes.html"><![CDATA[<h2 id="intro">Intro</h2>

<p>My light-hearted observation as an alliance leader is that EVE is 50% in-game, 50% discord, and 50% Google Sheets.
The idea is to create a solution for organizing my online gaming community that meets my users where they are: Discord and Google Sheets.</p>

<p>Getting the discord part, with its support for bots, has come naturally. But what’s often missed is an integration between Sheets and Discord. For example, editing data through Discord usually means it is no longer easily accessible, or up-to-date in sheets as the tendency is to move it to a database. Granted that has a lot of advantages and seems like the obvious thing to do, but it does have two main trade-offs that I think should give people pause:</p>

<ul>
  <li>Adding back the bulk CRUD operations that come easily in sheets once you’ve moved to SQL or another DB is added work and time.</li>
  <li>Google Sheets requires no maintenance and is free. This reduces my time spent as it/dev ops for my community.</li>
  <li>Everyone knows how to use sheets and build on them. It is easily extensible, and once again this requires no work or time from me.</li>
</ul>

<p>So with that in mind, the solution that I’ve been prototyping for managing my alliance, corporation, and pilots in the game uses Google Sheets as as the primary user interface. That means that it is also a primary store of data. The goal is for both Discord and Google Sheets to allow for the writing of data, though it doesn’t mean they need to have all the same features.</p>

<p>What follows is a walkthrough starting with scenarios that lead to the why and how of implementing an MVP of managing my EVE community’s data via Discord and Google Sheets.</p>

<h2 id="scenarios">Scenarios</h2>

<p>These are the scenarios that must be supported for this solution to fit my needs:</p>

<ul>
  <li>Bulk Update from Google Sheets by admins.</li>
  <li>Updates of individual rows by discord users via discord commands</li>
  <li>Invidiaul users can update their corp, clone, or otherwise a row of data from Discord via a command.</li>
  <li>Enable some users to perform analytics over the data.</li>
</ul>

<h2 id="requirements-and-constraints">Requirements and Constraints</h2>

<ul>
  <li>The data set is small. It can fit in memory with no chance of ever growing beyond that.</li>
  <li>The expected peak requests per second are fractional. But let’s go with 10 to leave a lot of headroom.</li>
  <li>Latency is important, but to a point. I want to avoid incidental latency from bad choices, but I won’t go so far as to write the whole thing in C++ or deploy it to multiple regions.</li>
  <li>Durability: If an update comes from Discord, it should not be lost on crash. There may, however, be a caveat here for conflict resolution to still roll back the update.</li>
</ul>

<h2 id="design">Design</h2>

<p>I’ll start by incrementally building the solution by adding the requirements one at a time, while minimizing the amount of work, and then see how it compares to a basic crud app.</p>

<p>Starting with just the requirement for discord users, we can start with a basic process and a store. Two boxes and a line, I won’t draw that. I will say that something like SQLite is enough for the store in this case, as it can be in process. And worth repeating that the entire data set can be held in memory.</p>

<p>Next, we add the requirement to bulk update and analyze data. Again, we can use a basic design here, with the store being any that persists beyond the discord bot, like a SQL instance, Cosmos, or a k/v store (none of the requirements have constrained that implementation yet). But, to keep the work to a minimum, and because I want to try it, we’re going to make the backend store a Google sheet. This gives us an admin view, authorization, and bulk edit, all for free. We can also do analytics over the sheet using… other sheets. And, if we update the sheet live, all that propagates as well.</p>

<p>How does this affect latency and other details?</p>
<ul>
  <li>One, I’m going to assume, (though I have not tested it), that Google Sheets does add more latency than the other options, especially on read.  </li>
  <li>Two, Google Sheets has no atomic updates using optimistic concurrency. It has no etags in its API.</li>
</ul>

<p>We’ll deal with two much later. But with one, we’re going add a second store to handle read/writes, and then sync them to sheets.</p>

<p>So, what have we arrived at? If you imagine a basic CRUD app, we’ve removed the (likely) js front end, replaced it with sheets, and added a sync to the primary store to compensate. Both arrangements have an additional Discord bot client to handle the Discord aspects. The bet is that using sheets but having the sync is cheaper in development and maintenance than a js front end. I’ve left the implementation of the store open-ended until the next section.</p>

<p>A diagram of the result, with some additional details of the interactions:</p>

<pre class="mermaid">
graph LR;
 User --Slash commands--&gt; Discord
 Discord --API Gateway--&gt;DiscordBot;
 DiscordBot--Discord Client--&gt;Discord;
 DiscordBot &lt;--??--&gt;Store;
 GoogleSheets &lt;--sync--&gt; Store;
 Admin &lt;--⌨️,👀--&gt; GoogleSheets
 Admin --Slash commands--&gt; Discord
</pre>

<p>An update from Discord, to show that updates to Google Sheets are eventually consistent, not atomic.</p>

<pre class="mermaid">
sequenceDiagram
 Discord User-&gt;&gt;Discord Bot: Update Corp
 Discord Bot&lt;&lt;-&gt;&gt;Data Store: Update
 Discord Bot-&gt;&gt;Discord User: Update Complete
 Data Store&lt;&lt;-&gt;&gt;Google Sheets: Sync
</pre>

<p>We have potentially created an issue. We’ve traded off consistency for reduced latency. It is now possible to temporarily see different results in sheets than in Discord until the sync happens. It is also possible, without a lot more work,  to create conflicts if both are updated simultaneously.</p>

<p>For the moment, I’m going to say that this is ok. I say this given the context of the small number of users, and the available resources (that is to say: none), and that the users themselves can easily fix any issues that come up from conflicts. The next step is still to create an MVP and see if this is a problem at all in this context. If this is for a Fortune 500 company with a well-funded dev team, my answer would be very different. And that’s the point.</p>

<h2 id="mvp">MVP</h2>

<p>For the MVP, the thing I choose to focus on is my admin users and getting the correct data in bulk. That means: Discord is read-only to start with. This avoids needing a sync, for the moment. However, to choose the store I do need to plan ahead.</p>

<p>For the choice of the store:
It looks a lot like a cache. So Redis is an option. It can also be an SQL instance. Revisiting the requirements: durability is important here, and it needs to sync updates back to the Google Sheets store. With that in mind, I came up with two options that kept the maintenance down by avoiding standing up a Redis cluster or SQL instance:</p>

<p>1) The process running the DiscordBot uses Raft.Next consensus to build an in-memory distributed K/V store. <a href="https://dotnet.github.io/dotNext/features/cluster/raft.html">https://dotnet.github.io/dotNext/features/cluster/raft.html</a>. Ensures availability, and that no updates are lost (assuming multiple nodes)</p>

<p>2) Cosmos. Free for this scale, and has a change feed processor to ensure all updates are eventually processed. And this almost certainly uses a consensus algorithm, like Raft, under the hood (as does SQL in replication scenarios).</p>

<p>I got fairly far with 1), but decided that if I ever wanted anyone to help me 2) is the more maintainable option. Finally, the sync from sheets to cosmos, for the MVP, is triggered via discord command. So for the MVP we have:</p>

<pre class="mermaid">
graph LR;
 User --Slash commands, read-only--&gt; Discord
 Discord --API Gateway--&gt;DiscordBot;
 GoogleSheets -.Via Bot.-&gt; Cosmos
 DiscordBot--Discord Client--&gt;Discord;
 DiscordBot &lt;--Cosmos client--&gt;Cosmos;
 DiscordBot &lt;--Sheets client--&gt;GoogleSheets;    
 Admin &lt;--⌨️,👀--&gt; GoogleSheets
 Admin --Slash commands, read-only--&gt; Discord    
</pre>

<p>I will end here for now. The next updates will be about what this MVP showed, and iterating the design from there.</p>

<h2 id="disclaimer">Disclaimer:</h2>
<p>Not generated using AI. Edited with the help of Grammarly.</p>

<h2 id="acknowledgment">Acknowledgment</h2>
<p>Mermaid integration achieved with help from: <a href="https://jackgruber.github.io/2021-05-09-Embed-Mermaid-in-Jekyll-without-plugin/">https://jackgruber.github.io/2021-05-09-Embed-Mermaid-in-Jekyll-without-plugin/</a> and <a href="https://mermaid.js.org/config/usage.html#simple-full-example">https://mermaid.js.org/config/usage.html#simple-full-example</a></p>

<p><a href="https://www.buymeacoffee.com/sarpedontdw" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174" /></a></p>]]></content><author><name>SarpedonTD</name></author><category term="KRAKEN" /><category term="Sheets" /><category term="Discord" /><summary type="html"><![CDATA[Intro]]></summary></entry><entry><title type="html">Rivals Legion Maps</title><link href="/2022/11/20/Legion-Maps.html" rel="alternate" type="text/html" title="Rivals Legion Maps" /><published>2022-11-20T00:00:00+00:00</published><updated>2022-11-20T00:00:00+00:00</updated><id>/2022/11/20/Legion-Maps</id><content type="html" xml:base="/2022/11/20/Legion-Maps.html"><![CDATA[<h2 id="background">Background</h2>
<p>Rivals is a Command and Conquer game for mobile that came out a few years ago. I’m going to skip right past discussing that as a concept. It does have a tight-knit community of players. Rivals was the first game I really used Discord for interacting with other players. Thanks to Davo (another member of the Rivals community) it also became the first scenario I ended up writing a Discord bot for. This was early Discord bot days. So before intents, and before slash commands.</p>

<p>The simplest use case for a bot for Rivals is to show a map. Rivals has 48 maps you can play on. And of ccourseyou can challenge other players to a game. So it makes sense to want to look up the map ahead of time.</p>

<p>This is probably my favorite map (the only map I’ve beaten 13lade on, that’s a story for another time):
<img src="http://davoonline.com/rivals/Bot/maps/orbit.png" alt="Orbit" /></p>

<p>While Rivals has a random map feature in game it’s revealed at the last moment when a match starts. For tournaments it helps to pick a map at random, but ahead of time so that both players know what map to expect and can pick thier decks accordingly. Finally, despite what looks like on the surface to be a small map and therefore simple, it turns out there is a lot of nuance to maps. Some are not at all fun to play on and poorly balanced. I’m looking at you:</p>

<p><img src="http://davoonline.com/rivals/Bot/maps/canalrow.png" alt="Canal Row" /></p>

<h2 id="features">Features</h2>
<p>The basic use case is perfect for getting started with a bot. There’s no calculation, no state. The functionality is simple: a command that just picks a map at random, and has no arguments. In the first days that would be a “~map” command, that would go through the message received handler. Legion uses the “~” prefix to identify commands. This of course means for a simple use case this bot would need to read message content. Uou could limit it to a specific channel at least. This was the typical model for those of us that were vigilant about security.</p>

<p>I looked over the code yesterday, and I forgot just how much more there can be to it after the basic use case. I completely forgot that the bot supports localization in Russian and German (translations provided to me by members of the community). As one small touch, for example, it responds to ~karte for German as well as ~map.</p>

<p>Beyond picking a map, players often want to see a particular map ahead of time. So you want something like “~map orbit”. This immediately brings up the observation that bots start to make Discord look like a command like shell, but is missing auto complete and tool tips. Discord’s App Commands helped address that once they were introduced.  But until then the bot needed a way to get the map that was closest to what was type. Cause if you typed ~map &lt;name&gt; you clearly wanted a map, so it should always return the best guest at a map even if there was no perfect match.</p>

<p>Finally, for tournaments, or just for fun, you could create and store map groups. It’s a named list of maps that could also be used to supply the valid maps used for a tournament.</p>

<h2 id="edit-distance">Edit Distance</h2>

<p>For making sure the command always returned a map I started with https://www.nuget.org/packages/Fastenshtein. Given an input it would return the closest localized map name. One trick it would do is remove “the”, or “das” and “der” if in German, from the map name as this would create some noise in the edit distance. Looking at the code now I found a small bug.  I used the wrong normalizer for one of the languages so the wrong words would get removed before looking up edit distance.</p>

<p>Because the command ALWAYS returns a map now based on edit distance, there was some brief fun around looking up what maps a particular player name would result in. For example, “Sarpedon” has the smallest edit distance of all the maps to “Caged In”</p>

<h2 id="using-a-slash-command">Using a Slash Command</h2>
<p>So now there are app commands, and “map” makes for a great global command. This is the snippet for adding the command:</p>

<pre><code class="language-C#">        SlashCommandProperties guildMessageCommand = new SlashCommandBuilder()
            .WithName(CommandName)
            .WithDescription("Picks a map at random")
            .WithDMPermission(true)
            .WithDefaultMemberPermissions(GuildPermission.SendMessages)
            .Build();

await client.BulkOverwriteGlobalApplicationCommandsAsync( new ApplicationCommandProperties[] {guildMessageCommand});
</code></pre>
<p>This is a snippet of the handler:</p>
<pre><code class="language-C#">    private Task _client_SlashCommandExecuted(SocketSlashCommand arg)
    {
        if (arg.CommandName == CommandName)
        {
            return arg.RespondAsync(embed: ShowMap(this.maps[R.Next(this.maps.Count)]).Build());
        }
        return Task.CompletedTask;
    }

    private static EmbedBuilder ShowMap(MapValues map)
    {
        EmbedBuilder builder = new EmbedBuilder();
        builder.WithTitle(MapLookup.GetMapName(map));
        builder.WithImageUrl(MapLookup.GetUrl(map));
        return builder;
    }
</code></pre>

<p>And using application commands makes it really easy to add this command to a server: 
https://discord.com/api/oauth2/authorize?client_id=1043771859159756840&amp;scope=applications.commands</p>

<p>There is some boilerplate to a bot, however. For example:</p>
<ul>
  <li>Logging</li>
  <li>Storing the secret</li>
</ul>

<p>As part of creating this command I think I have a good template I can leverage in the future, and share.</p>

<h2 id="options">Options</h2>
<p>I might be able to make it easier to look up a name by showing a list of all names as part of the slash command. Discord commands can be created with pre-set options for the arguments. Unfortunately the limit to options is 25. But there is grouping, so I can break down the 48 maps into some grouping (maybe alphabetically) so each group would have less than 25 options.</p>

<p>Finally, there is an auto-completion capability as well: https://discord.com/developers/docs/interactions/application-commands#autocomplete. Lots of way to improve the simple command with Discord features.</p>

<p>So that’s a brief random walk through bots, maps, Legion, and Rivals. The intent is there will be a few more Legion related posts!</p>

<p>Visit 13lade’s Twitch: <a href="https://www.twitch.tv/13lade">https://www.twitch.tv/13lade</a>
There’s a Rivals tournament facilitated by Legion today on this channel!</p>

<p>Please support Legion and other tech: <a href="https://www.buymeacoffee.com/sarpedontdw" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174" /></a></p>]]></content><author><name>SarpedonTD</name></author><category term="Discord" /><category term="Legion" /><summary type="html"><![CDATA[Background Rivals is a Command and Conquer game for mobile that came out a few years ago. I’m going to skip right past discussing that as a concept. It does have a tight-knit community of players. Rivals was the first game I really used Discord for interacting with other players. Thanks to Davo (another member of the Rivals community) it also became the first scenario I ended up writing a Discord bot for. This was early Discord bot days. So before intents, and before slash commands.]]></summary></entry><entry><title type="html">Guest Wieschie On Ocr</title><link href="/2022/11/12/Guest-Wieschie-On-OCR.html" rel="alternate" type="text/html" title="Guest Wieschie On Ocr" /><published>2022-11-12T00:00:00+00:00</published><updated>2022-11-12T00:00:00+00:00</updated><id>/2022/11/12/Guest-Wieschie-On-OCR</id><content type="html" xml:base="/2022/11/12/Guest-Wieschie-On-OCR.html"><![CDATA[<p>This is guest written by <a href="https://discord.com/users/81774426697248768">[RTI] Wieschie</a>. He has written the OCR component of VOID’s PPK workflow.
Here is him describing highlights of solving that!</p>

<h1 id="killmail-ocr-development">Killmail OCR Development</h1>

<p>Killmails are many things in Eve - they can be bragging rights, ways to track enemy movements, tactics, and fits, or just <a href="https://zkillboard.com/kill/104468891/">a mess that everyone wants to gather around and stare out</a>. Eve Online provides a killmail API that allows players to develop a rich set of 3rd party tools that have access to all of this data. Unfortunately, the best we can get in Eve Echoes is player-submitted screenshots. Screenshots are nice to share with your friends, but it’s impossible to glean anything from them on a larger scale. So the solution has become creating software that can read these screenshots and extract the data we need.</p>

<p>I have set out to build a free and open-source killmail parser for Eve Echoes.</p>

<p>Many other parsers are using machine vision APIs from cloud services to pull text from images. This is fast, easy, supports more types of scripts and languages, and can be very accurate. However, it costs money and locks you into a specific cloud provider. We want something that we can tinker with and reliably self-host without worrying if a service will increase in price or a third party bot will go down.</p>

<h2 id="using-tesseract">Using Tesseract</h2>

<p>Tesseract is a free Optical Character Recognition (OCR) engine. You feed it a picture and it spits out text. There are 3 basic steps when processing an image using OCR:</p>

<ol>
  <li>Image Pre-processing</li>
  <li>OCR Execution</li>
  <li>Text Post-processing</li>
</ol>

<h3 id="image-pre-processing">Image Pre-processing</h3>

<p>Tesseract uses an LSTM neural network to transform images into text. There’s a bunch of fancy math behind it, but practically it means that you need a model to tell the software how to translate an image of text into the actual characters it contains. OCR is widely used to digitize scans of printed text, and so these models are trained to recognize black text on a white background. This is nearly the exact opposite of what we get in killmails. Most text is white or lightly saturated colors, on a dark or patterned background. Cleaning this up first helps us get better results.</p>

<p>Here’s a killmail header. The background is fairly clean, but the color scheme is not quite right.</p>

<p><img src="https://i.imgur.com/mbzAuMi.png" alt="base-header" /></p>

<pre><code class="language-python"># convert our image to grayscale
img = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
# invert the colors so the text is black
img = cv2.bitwise_not(img)
</code></pre>

<p>And viola!</p>

<p><img src="https://i.imgur.com/MJWg3Ao.png" alt="processed-header" /></p>

<p>Here’s a more complex example. The victim name is pure white text on a background with a space pattern that changes colors based on the type of ship that was killed. This variation in the background can really confuse OCR.</p>

<p><img src="https://cdn.discordapp.com/attachments/1028446254864277514/1039974147108904960/image.png" alt="base-clone" /></p>

<pre><code class="language-python"># Convert our image from BGR (Blue-Green-Red) to HSV (Hue-Saturation-Value) to make this kind of filtering easier.
cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
# Define an HSV range that the text we want to parse falls inside, while the background falls outside
low = (0, 0, 180)
high = (255, 80, 255)
# inRange returns a 255 if the source pixel is inside the HSV window and 0 when outside.  This will result in white text and a black background
img = cv2.inRange(img, low, high)
# invert our image to get black text on a white background
img = cv2.bitwise_not(img)
</code></pre>

<p>With a little cleanup, we get an image that looks a little rougher but is very straightforward to identify what is text and what is not. This is called binarization - reducing an image to only 2 values - black and white.</p>

<p><img src="https://cdn.discordapp.com/attachments/1028446254864277514/1039974146744008815/image.png" alt="processed-clone" /></p>

<h3 id="ocr-execution">OCR Execution</h3>

<p>Using Tesseract is really straightforward - you give it an image, and it gives you text back. There are a few knobs you can tweak to get better results.</p>

<pre><code class="language-python">import pytesseract as pt
pt.image_to_string(img, lang="eng-shentox", config="--psm 7")
</code></pre>

<p>First, Tesseract offers script detection - given an image of text, it will attempt to tell you the written script (such as Latin, Han, Hangul, etc.) that the text is using. This allows us to then choose the best model to parse the text.</p>

<p>Tesseract has a variety of runtime config options, including page-segmentation mode. This is essentially how Tesseract will chop up the image into pieces when looking for text characters. Trying different options can yield better results.</p>

<p>Tesseract is optimized to recognize whole words from a dictionary. Because much of the text we are trying to recognize is either nonsense (character names) or non-word phrases, we can disable the default dictionary or substitute it with our own list of Eve terms that we expect to see.</p>

<p>In addition, you can ask Tesseract for detailed information about the locations that text was identified, and how confident it is that the text was correctly transcribed. It may be useful in the future to look at confidence levels and attempt different methods to see if a more accurate result can be achieved.</p>

<h3 id="text-post-processing">Text Post-processing</h3>

<p>Now that we’ve processed our images, we’re left with a bunch of text.</p>

<p>Corp tags are one of more frequent culprits for errors. Tesseract often returns text that looks like <code>[ABCJPilot</code> or <code>(ABC]Pilot</code>. These can be cleaned up with some simple find and replace operations.</p>

<p>System names can be difficult to get exactly right. The good news is that we know what all systems in the game are named. So instead of returning the raw text from the image, we can pick the closest-matching actual system and use that value.</p>

<pre><code class="language-python">from thefuzz import process
# a list of all system &gt; constellation &gt; region pairs in New Eden
systems = load_systems()
# the raw location parsed from the image.  There's two incorrectly identified characters here
location = "E1UU-B &lt; 3WN-IT &lt; Esoteria"
# perform a fuzzy text lookup to recieve the closest match and its score
actual_location, location_confidence = process.extractOne(location, systems)
print(f"{actual_location}, {location_confidence}")
&gt; "E1UU-3 &lt; 3WN-1T &lt; Esoteria", 92
</code></pre>

<h2 id="fine-tuning-tesseract">Fine Tuning Tesseract</h2>

<p>Tesseract has a high level of accuracy on common fonts that you would expect to see in print. It does fairly well with screenshots of text, but the model can get tripped up on specific characters or combinations with a new font that it hasn’t trained on. Is that <code>H</code> or <code>|-|</code>? The brackets in corp tag like [ABC] can be confused with any of these characters: <code>Il|i1)}J</code>.</p>

<p>Creating a model requires a set of training data. This data consists of questions (images containing text) and answers (the exact coordinates of every single character and its value). This can be a very labor-intensive process if you are training on photographs. Some poor person has to draw a <a href="https://raw.githubusercontent.com/A2K/jTessBoxEditor/master/screenshot.png">pixel-perfect box</a> around every character separately and type in the correct value.</p>

<p>Thankfully, if you have a copy of the font you are trying to recognize, there’s an easier way. Tesseract provides scripts that will generate images and boxes given any sample text and a font. This data can then be used to refine the existing models - taking most of the performance of the original, but cleaning up some of the edge cases. This process is <a href="https://tesseract-ocr.github.io/tessdoc/tess4/TrainingTesseract-4.00.html#creating-training-data">described in detail in their documentation</a>. Eve Echoes uses the <a href="https://emtype.net/fonts/shentox">Shentox Light font</a>.</p>

<hr />
<p>You can support VOID’s PPK and other tech: <a href="https://www.buymeacoffee.com/sarpedontdw" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174" /></a></p>]]></content><author><name>SarpedonTD</name></author><category term="VOID" /><category term="Discord" /><category term="Guest" /><category term="PPK" /><summary type="html"><![CDATA[This is guest written by [RTI] Wieschie. He has written the OCR component of VOID’s PPK workflow. Here is him describing highlights of solving that!]]></summary></entry><entry><title type="html">IT for games</title><link href="/2022/11/10/Game-IT.html" rel="alternate" type="text/html" title="IT for games" /><published>2022-11-10T00:00:00+00:00</published><updated>2022-11-10T00:00:00+00:00</updated><id>/2022/11/10/Game-IT</id><content type="html" xml:base="/2022/11/10/Game-IT.html"><![CDATA[<p>A lot of the tools we need to organize players are the same tools large enterprise use. Of course the cheapest (read: free) wins. 
That’s why we use Google sheets, Discord. But I do wish we could use things like Airtable sometimes for example. Most of them are in $5 or more <em>per user</em> <strong>per month</strong>* range. 
There’s just no budget for that in an alliance of 1000+. That would be a lot more coffee we’d need ;)</p>

<p>But the free stuff is pretty good. And even better when you actually use it effectively. Which is why I was so embarrassed it took me <em>THIS</em> long to figure out a simple way to organize VOID google sheets.
We use a lot of google sheets. That means a lot of permissions. In EVE we want secret things to remain secret, so not having open permissions is important. 
And people come and go, so updating permissions is important and also time consuming if you have to do it for every sheet every time someone joins or leaves. 
Another issue was having to get myself, as Executor, added to just about everything.</p>

<p>Google Drive is all it took. I don’t know why I didn’t over a year ago. The trick is folder and files inherit the permissions of their parent folder. So you can effectively edit many permissions at once. 
This is the exact screenshot my post to my alliance had when I described what we were going to do. It seemed pretty obvious in retrospect.</p>

<p><img src="https://cdn.discordapp.com/attachments/999577232190226483/1028823710246907984/unknown.png" alt="Google Drive Organization" /></p>

<p>As a bonus point it was then very easy to add a Google service account later to help facilitate Discord to Google Sheet integrations.</p>

<p>It’s the little things!</p>

<p><a href="https://www.buymeacoffee.com/sarpedontdw" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174" /></a></p>]]></content><author><name>SarpedonTD</name></author><category term="VOID" /><summary type="html"><![CDATA[A lot of the tools we need to organize players are the same tools large enterprise use. Of course the cheapest (read: free) wins. That’s why we use Google sheets, Discord. But I do wish we could use things like Airtable sometimes for example. Most of them are in $5 or more per user per month* range. There’s just no budget for that in an alliance of 1000+. That would be a lot more coffee we’d need ;)]]></summary></entry><entry><title type="html">On Costs</title><link href="/2022/10/22/On-Costs.html" rel="alternate" type="text/html" title="On Costs" /><published>2022-10-22T00:00:00+00:00</published><updated>2022-10-22T00:00:00+00:00</updated><id>/2022/10/22/On-Costs</id><content type="html" xml:base="/2022/10/22/On-Costs.html"><![CDATA[<p>As I was drafting a post in my head about the storage schemas and architecture used in Orphyx and Legion I realized I would first have to talk about cloud costs. A lot of the constraints on the schema are informed by the infrastructure we chose to use.  This is not meant to be exhaustive or definitive. It is definitely biased by my own prior experience with various solutions. I’ve probably missed cost savings, and if you know of any please DM me in Discord: <a href="https://discord.com/users/1025270667970609182">https://discord.com/users/1025270667970609182</a></p>

<p>There are primarily three things I am optimizing for:</p>
<ul>
  <li>Monthly dollar cost of running the bots</li>
  <li>Time cost of maintaining the solution.</li>
  <li>Time cost to implement the solutions. This is incurred once.</li>
</ul>

<p>From a technical point of view : 
I will seek to make reasonable choices between cost and reducing bot latency to respond to users. This, for example, may inform choices between using a VM or using Lambdas as the main way of interacting with Discord. The VM has much more predictable latency. The details of that are probably a different post.</p>

<p>I will not seek to make my solution infinitely scalable. We have a small number of users, so I can do things like hold all of them in memory easily if I need to and not feel guilty about it.<br />
Things like Disaster Recovery are, however, important to me.</p>

<p>Ok, so breaking it down, we have the usual suspects:</p>

<ul>
  <li>Compute cost of a background worker. Example is the bot responding to Discord events.</li>
  <li>Cost of hosting a front end</li>
  <li>Cost of storage.</li>
</ul>

<p>There are other costs, like domain hosting, but there’s very little if any interaction with the code and tech choices which is why the post is focussed on cloud provider prices.</p>

<p>Finally, I will ignore any free-trial options or credits. I want to know what I will be paying per month in the steady state.</p>

<h2 id="baseline-vm">Baseline VM</h2>

<p>So first, you can get both a front end and storage for $5 a month by getting a VM from Linode or Digital Ocean. That’s for 1GB and 1 shared CPU. Then put a web server and mysql on it. There are some drawbacks here: 
You are responsible for backup and disaster recovery. (Something we’ve already had to deal with once). It’s up to you to make sure you can rebuild the VM and not have lost any data. 
We do use this to host our alliance’s credit system, and it’s already pushing the resource limits of one cheap node despite not having anything complex in what it does. 
And interestingly, hosting the SQL and site on the same site led to some questionable design shortcuts which made it harder to extend the solution. This wasn’t my choice (we inherited the solution), but it was an observation I wanted to note.</p>

<p>You can get similar offers free from Google or Oracle. The 0$ to $5 options are a  good baseline to compare against.</p>

<h2 id="front-end">Front End</h2>

<p>Both Google and Azure have free web hosting:</p>

<p><a href="https://azure.microsoft.com/en-us/pricing/details/app-service/linux/#pricing">https://azure.microsoft.com/en-us/pricing/details/app-service/linux/#pricing</a>
<a href="https://cloud.google.com/free/docs/free-cloud-features#app-engine">https://cloud.google.com/free/docs/free-cloud-features#app-engine</a></p>

<p>While it is also free, the real win for me here is the potential of less maintenance, I don’t need to manage the VM in any way.</p>

<p>This only saves money if I have a place to get storage for $5 or less.</p>

<h2 id="storage">Storage</h2>

<h3 id="sql">SQL</h3>
<p>The basic SQL options exist in AWS, Azure, and GCP. The lowest I could estimate was probably $7 a month.</p>

<h3 id="cosmos">Cosmos</h3>
<p>There’s a free tier of Cosmos now. Legion actually uses Cosmos DB. It pre-dates the free tier and I will have to move all the data to take advantage of this (kind of annoyed by this actually). A future blog post, maybe. Here’s the free tier: <a href="https://learn.microsoft.com/en-us/azure/cosmos-db/free-tier#best-practices-to-keep-your-account-free">https://learn.microsoft.com/en-us/azure/cosmos-db/free-tier#best-practices-to-keep-your-account-free</a></p>

<p>I do have some reservations here as I’m not confident that Legion and Orphyx will predictably stay in the RU limit. But it is actually a very tempting offer because it is also easy to code against since it has something that SQL has, but the next option lacks: Secondary Indexes.</p>

<h3 id="nosql">NoSql</h3>
<p>Finally, there’s the NoSQL stores. Dynamo, Azure Table and Firestore or other equivalents. 
Orphyx is actually currently on this. And it runs me $0.01 a month right now.  Google and AWS would have similarly priced (free) solutions. The major disadvantage is that without features of SQL that even Cosmos has there are some extra design constraints to navigate that often result in extra code you have to carefully write. And that will absolutely be the topic of future posts.</p>

<h2 id="summary">Summary</h2>
<p>Currently where we are at is that we have one lineode for $5, and actually one Windows Azure VM with a reservation to bring it to $11 a month. The linode should eventually melt away into a free front end option and one of the storage options. 
Legion uses Cosmos, and Orphyx uses table storage. After moving Legion to the free tier of Cosmos or moving Legion to a NoSQL option the price of that will also be in range of a few cents a month. Maintenance costs and single points of failure (a VM) are minimized.</p>

<p><a href="https://www.buymeacoffee.com/sarpedontdw" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174" /></a></p>]]></content><author><name>SarpedonTD</name></author><category term="Legion" /><category term="VOID" /><category term="cloud" /><summary type="html"><![CDATA[As I was drafting a post in my head about the storage schemas and architecture used in Orphyx and Legion I realized I would first have to talk about cloud costs. A lot of the constraints on the schema are informed by the infrastructure we chose to use. This is not meant to be exhaustive or definitive. It is definitely biased by my own prior experience with various solutions. I’ve probably missed cost savings, and if you know of any please DM me in Discord: https://discord.com/users/1025270667970609182]]></summary></entry><entry><title type="html">On Discord and PPK</title><link href="/2022/10/20/On-Discord-and-PPK.html" rel="alternate" type="text/html" title="On Discord and PPK" /><published>2022-10-20T00:00:00+00:00</published><updated>2022-10-20T00:00:00+00:00</updated><id>/2022/10/20/On-Discord-and-PPK</id><content type="html" xml:base="/2022/10/20/On-Discord-and-PPK.html"><![CDATA[<p>First, for anyone not familiar with the acronym, in EVE PPK is Pay Per Kill.</p>

<p>EVE runs on Google Sheets and Discord. I’ll get more into Discord particulars later. But PPK systems are one reason. 
A central part of EVE Echoes is a kill mail. Here’s one for reference:</p>

<h2 id="overview">Overview</h2>
<p><img src="https://cdn.discordapp.com/attachments/758102282276700232/908505170760056832/unknown.png" alt="A Kill Mail" /></p>

<p>The game, unlike EVE Online, does not have a KM API. Most alliances therefore have to implement some work around. And Discord is often central to that work around.
It starts with OCR of the KM above when shared. I won’t go into that here, because that’s not  a part I am familiar with. Though I thank the devs that do that portion. 
Suffice it say after OCR you have the various fields from the image available. Things like:</p>
<ul>
  <li>The victim and victim’s corp.</li>
  <li>The value.</li>
  <li>The Final Blow and Top Damage participants.</li>
</ul>

<p>The typical flow for building an alliance PPK program goes like this:</p>

<ol>
  <li>User grabs an image of the KM from the game, and posts an often cropped version of it into a Discord channel dedicated to kill mails.</li>
  <li>A bot picks up the image, runs it through OCR, and outputs the information parsed into some schematized store.</li>
  <li>Some analytics are run and output somewhere for alliance members to see.</li>
</ol>

<h2 id="ppk-events">PPK Events</h2>
<p>For VOID, I wanted to expand on what PPK means. Typically PPK is for final blow. The person gets some % of the kill in ISK back. 
So in VOID’s system, I have a KM still, but also added a separate PPK event. A typical KM already has three PPK events:</p>
<ul>
  <li>Damage</li>
  <li>Final shot</li>
  <li>Participants.</li>
</ul>

<p>All of these we can give PPK for. On top of that, things that don’t appear on the kill mail, but we can add as PPK events:</p>
<ul>
  <li>The FC (Fleet Commander)</li>
  <li>All Logi (EVE healers)</li>
  <li>Tackle, and interdiction</li>
  <li>Scouts</li>
</ul>

<p>This will look like a DKP system for anyone familiar with DKP (Dragon Kill Points) from other MMOs. But the DKP is backed by ISK, which is obtained in-game.</p>

<h2 id="pilot-workflow">Pilot Workflow</h2>

<p>The workflow, from a Pilot’s perspective, looks like this:</p>

<ol>
  <li>Get a KM</li>
  <li>Post the KM to dedicated Discord channel</li>
  <li>Other pilot’s involved in the KM can then added their own PPK events. FCs, tackle, can add PPK events through commands or buttons createed by a Discord bot. 
Because its in a visible channel there’s very little risk of someone lying.</li>
  <li>A summary is created run at some point, and through some manual method the pilots are given the PPK for the week/month or for each individual PPK event through a ticket bot.</li>
</ol>

<h2 id="storage-for-the-workflow">Storage for the Workflow</h2>

<p>For design of storage, most systems have an OLTP and OLAP system. 
For VOID Azure Table backs the initial ingress of KM and PPK events. I’ll leave the schema for a different time. 
And we creatively use Google Sheets as the OLAP store. 
The reason for these two picks is mainly due to cost and ease of maintenance. 
Each PPK event is eventually written to a google sheet dedicated to a specific month for all VOID, and a separate sheet for the pilot’s corp. This allows corp’s to build thier own unique systems without having to re-create all the other parts.</p>

<h2 id="adding-ppk-events-via-discord">Adding PPK Events via Discord</h2>

<p>For steps 3 in <a href="#pilot-workflow">Pilot Workflow</a> the Discord bot listens for KMs, creates a row for the KM, then creates PPK buttons. Note that PPK is assigned to an in-game clone, not a Discord user.</p>

<pre><code class="language-C#">// Wire up events to process messages and button clicks
_client.MessageCommandExecuted += handlers.ClientOnMessageCommandExecuted;
_client.ButtonExecuted += handlers.ClientOnButtonExecuted;

   internal Task ClientOnMessageCommandExecuted(SocketMessageCommand messageCommand)
    {
        // Code omitted to check certain things about the buttons. The killMailId is parsed from the message.
        
        this._logger.Log($"Creating buttons.");
        ComponentBuilder builder = this.CreatePpkButton(killMailId);
        messageCommand.RespondAsync($"Add a PPK Event to Killmail ID [{killMailId}] [&lt;{messageCommand.Data.Message.GetJumpUrl()}&gt;]:", components: builder.Build());
        return Task.CompletedTask;
    }

 internal ComponentBuilder CreatePpkButton(string killMailId)
    {
       // Note the use of the custom Id to plumb through the killmail Id.
        return new ComponentBuilder()
            .WithButton("PARTICIPANT", $"{KillMailParticipantType.PARTICIPANT}-{killMailId}", ButtonStyle.Primary)
            .WithButton("TACKLE", $"{KillMailParticipantType.TACKLE}-{killMailId}", ButtonStyle.Danger)
            .WithButton("DICTOR", $"{KillMailParticipantType.DICTOR}-{killMailId}", ButtonStyle.Danger)
            .WithButton("LOGI", $"{KillMailParticipantType.LOGI}-{killMailId}", ButtonStyle.Success)
            .WithButton("SCOUT", $"{KillMailParticipantType.SCOUT}-{killMailId}", ButtonStyle.Danger)
            .WithButton("FC", $"{KillMailParticipantType.FC}-{killMailId}", ButtonStyle.Danger);

    }
    
    internal Task ClientOnButtonExecuted(SocketMessageComponent messageComponent)
    {
        // Get the KM Id, and the PPK Event selected by parsing the custom Id
        string dataCustomId = messageComponent.Data.CustomId;
        this._logger.Log($"{messageComponent.User.Mention} has clicked the button with id [{dataCustomId}]");
        DiscordId pilotId = (DiscordId)messageComponent.User.Id;
        string[] result = dataCustomId.Split("-");
        string type = result[0];
        string killMailId = result[1];

        this._logger.Log($"[{type}]-[{killMailId}] button pressed.");
        
        // Look up the clones to assign PPK to
        IEnumerable&lt;Clone&gt; clones = this._cloneRepository.GetClonesByDiscordId(pilotId);
        IEnumerable&lt;Clone&gt; enumerable = clones.ToList();

        if (enumerable.Count() == 1)
        {
            Clone clone = enumerable.Single();
            this._logger.Log($"Pilot with single clone {clone.cloneName} being added to PPK");
            
            // Store the PPK event
            return this.AddPpkForClone(messageComponent, clone, type, killMailId, pilotId);
        }
     
     // Some more code ommited for brevity to deal with when the user has more than one registered clone.    
    }
</code></pre>

<h2 id="further-discord-commands">Further Discord Commands</h2>

<p>Because things can go wrong, you also need commands for the following locked to certian discord user considered admins:</p>
<ul>
  <li>Re-parse and re-create the buttons for a KM message.</li>
  <li>Manually override any values parsed wrong.</li>
  <li>Manually remove or add a PPK event on behalf of another pilot.</li>
</ul>

<p>So that’s a highlevel overview of VOID’s PPK Events, with some details about selecting PPK Events via Discord.</p>

<p><a href="https://www.buymeacoffee.com/sarpedontdw" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174" /></a></p>]]></content><author><name>SarpedonTD</name></author><category term="VOID" /><category term="Discord" /><summary type="html"><![CDATA[First, for anyone not familiar with the acronym, in EVE PPK is Pay Per Kill.]]></summary></entry><entry><title type="html">Adding checkboxes to Google sheets, because EVE</title><link href="/2022/10/14/Google-Sheet-Checkboxes.html" rel="alternate" type="text/html" title="Adding checkboxes to Google sheets, because EVE" /><published>2022-10-14T00:00:00+00:00</published><updated>2022-10-14T00:00:00+00:00</updated><id>/2022/10/14/Google-Sheet-Checkboxes</id><content type="html" xml:base="/2022/10/14/Google-Sheet-Checkboxes.html"><![CDATA[<p>This post will be on a specific detail of something I’ve recently done. I do plan on putting some over-arching design posts up as well soon.</p>

<p>In EVE a lot of things are kept secret for what is called op-sec reasons, and there will be a lot of that in this blog. So I won’t go into the why of it.
Another thing about EVE Echoes is that I call it the greatest text adventure game ever. As an executor, for better or worse, a great deal of my time spent playing EVE Echoes is spent on Discord. And for a lot of people another part of it is spent in google sheets.
EVE is sometimes referred to as a heavy weight Excel client. Which is why things like this happen: https://arstechnica.com/gaming/2022/05/eve-onlines-ms-excel-partnership-makes-spreadsheets-in-space-official/</p>

<p>And now, with the help of Discord bots, the two can meet. Discord bots will be a common topic here.
The basic thing I needed to do was get a bot to list something for a user. And I realized they were going to just dump it into google sheets anyways. So, it makes sense to just create that sheet.
On top of that, the user would want a column full of checkboxes to go with the data. Now that on its own is pretty easy to create in the UI:</p>

<p><img src="https://user-images.githubusercontent.com/52060413/195969961-9e0e63e7-ff20-4532-998e-b7dec1d69c31.png" alt="image" /></p>

<p>But it wasn’t so obvious from the API perspective. I had to do with any seasoned developer does. Search on google for stackoverflow answers.</p>

<p>Now our bot is in C#, and the google API docs don’t cater to C#.
This is the link I got the most use out of: 
<a href="https://webapps.stackexchange.com/questions/121502/how-to-add-checkboxes-into-cell-using-google-sheets-api-v4">https://webapps.stackexchange.com/questions/121502/how-to-add-checkboxes-into-cell-using-google-sheets-api-v4</a>.</p>

<p>A careful inspection of this will help once you know that you need a batch update for a conditional formatting request: 
<a href="https://developers.google.com/sheets/api/reference/rest/v4/spreadsheets/request">https://developers.google.com/sheets/api/reference/rest/v4/spreadsheets/request</a></p>

<p>And this is what the C# code looks like:</p>

<pre><code class="language-C#">            BatchUpdateSpreadsheetRequest request = new BatchUpdateSpreadsheetRequest
            {
                Requests = new List&lt;Request&gt;(
                new[]
                {
                    new Request
                    {
                        RepeatCell = new RepeatCellRequest()
                        {
                            Cell = new CellData
                            {
                                DataValidation = new DataValidationRule()
                                {
                                    Condition = new BooleanCondition
                                    {
                                        Type = "BOOLEAN",

                                    },
                                },
                            },
                            Range = new GridRange()
                            {
                                StartColumnIndex = 0,
                                EndColumnIndex = 1,
                                StartRowIndex = 1,
                                EndRowIndex = cellValues.Count,
                                SheetId = response.sheetId,
                            },
                            Fields = "dataValidation",
                        },
                    },
                })
            };
            SpreadsheetsResource.BatchUpdateRequest updateRequest = this._sheetsService.Spreadsheets.BatchUpdate(request, response.spreadsheet.SpreadsheetId);
            BatchUpdateSpreadsheetResponse? response = updateRequest.Execute();
</code></pre>

<p>Some notes:</p>
<ul>
  <li>To get one column of checkboxes, you need to pass a start and end index that are different by one. Setting them both to 0 in the above example yieds no checkboxes.</li>
  <li>They are unsurprisingly indexed at zero, despite the label of the first column being “1”.</li>
</ul>

<p>So there we go. This was a small piece that made the entire command much more user friendly. With a simple command the user got a ready-to-use sheet shared with them. 
And I made sure they didn’t even need to click a few times to add a checkbox column. As always, that last detail took the most time to do out of the entire feature.</p>

<p><a href="https://www.buymeacoffee.com/sarpedontdw" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174" /></a></p>]]></content><author><name>SarpedonTD</name></author><category term="sheets" /><category term="VOID" /><summary type="html"><![CDATA[This post will be on a specific detail of something I’ve recently done. I do plan on putting some over-arching design posts up as well soon.]]></summary></entry><entry><title type="html">Obligatory First Post</title><link href="/2022/10/12/Obligatory-First-Post.html" rel="alternate" type="text/html" title="Obligatory First Post" /><published>2022-10-12T00:00:00+00:00</published><updated>2022-10-12T00:00:00+00:00</updated><id>/2022/10/12/Obligatory-First-Post</id><content type="html" xml:base="/2022/10/12/Obligatory-First-Post.html"><![CDATA[<p>For almost as long as I’ve been playing video games I’ve also been writing hobby software to assist me or my friends in those games.</p>

<p>One of my more recent projects is a tournament bot, Legion, used to run tournaments on Discord for the mobile version of C&amp;C: Command &amp; Conquer: Rivals.
Rivals is how I met JadeXyan (co-creator of Legion),  and Captain Benzie. That set off a chain of unfortunate events that led to me becoming one of the executors of an alliance in Eve Echoes: The VOID Federation.</p>

<p>As part of VOID the software projects include:</p>
<ul>
  <li>Working on a team to maintain a credit system for the alliance members: <a href="www.voidcoin.app">www.voidcoin.app</a>.</li>
  <li>Creating another bot with JadeXyan caled Orphyx to maintain in-game character information for the 1000+ members of VOID.</li>
</ul>

<p>This is the first time I’ve created a blog. Though I’ve often wanted to create a blog, or a software blog. What has stopped me is I’ve never felt I had anything to write about. The blog itself a bit of learning for me. It was quickly created using github pages as well as the template and steps found here: 
<a href="https://chadbaldwin.net/2021/03/14/how-to-build-a-sql-blog.html">https://chadbaldwin.net/2021/03/14/how-to-build-a-sql-blog.html</a></p>

<p>I’ll be writing about these projects and delving into the technical details of them in this blog.</p>

<p>But really, the idea is to create semi-regular content as a way to shamelessly plug my buymeacoffee site. This is then used to then pay for the various tech tools of my current alliance, VOID.</p>

<p><a href="https://www.buymeacoffee.com/sarpedontdw" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174" /></a></p>]]></content><author><name>SarpedonTD</name></author><category term="Other" /><summary type="html"><![CDATA[For almost as long as I’ve been playing video games I’ve also been writing hobby software to assist me or my friends in those games.]]></summary></entry></feed>