Clever Geek Handbook

Élite: dangereux et CosmosDB

image



o7 cmdr!



Lors d'une chaude soirée de quarantaine, dans l'un des télégrammes de discussion sur Elite: Dangerous, une discussion a éclaté sur le sujet: quel type d'étoiles ont le plus souvent des planètes semblables à la Terre?



Le fait est que l'exploration des planètes est l'un des principaux mécanismes du jeu. Et dans la hiérarchie de l'utilité des planètes, celles qui ressemblent à la Terre sont tout en haut. Mais leur rareté est également assez élevée. Les commandants voulaient donc savoir: à quelles étoiles faire attention lors du déplacement sur la Voie lactée?



De cette discussion, tout un projet est né, que j'ai finalement enterré. Non, nous avons trouvé la réponse à la question posée avec. Mais je n'ai pas aimé le projet pour diverses raisons, et après quelques mois de procrastination, j'ai lancé la deuxième itération. Qu'est-il arrivé, ainsi que la réponse à la question posée - dans cet article.



introduction



Dans l'élite, une grande partie du gameplay se déroule, quoi que vous en pensiez, dans un navigateur avec une douzaine d'onglets ouverts. Nous ne discuterons pas si c'est juste ou non, mais, certainement, cela a son propre charme et beaucoup de gens l'aiment.



, , . , Frontier Developments . , .



? , Log- . .



() ( Windows):



C:\Users\User Name\Saved Games\Frontier Developments\Elite Dangerous\

. . .. PC ( ) — .



, , , . ? - , ( , , , , POI, etc).



PC:



Tool Commander
E:D Market Connector Otis B.
ED-Intelligent Boardcomputer Extension Duke Jones
edce-client Andargor
EDDI VerticalBlank, Hoodathunk, T'Kael
EDDiscovery Robby
Elite Log Agent John Kozak
Elite G19s Companion app MagicMau
Elite Virtual Assistant
Trade Dangerous + EDAPI orphu


. - . , . , , . , 400 (, ! ). , -.



, . ( , ). : , , (, ) .. , , , , — EDSM EDDB.



, , — . .





EDSM , . , , . , , — JSON 10. - - . , , - , . , ! .



image



% . 6 . .



, , . , , !



, , , - ( , ). EDSM, EDDB, etc. — EDDN. , , , .



, EDDN ? ...





EDDN



EDDN — :



The Elite: Dangerous Data Network is a system for willing Commanders to share dynamic data about the galaxy with others.

By pooling data in a common format, tools and analyses can be produced that add an even greater depth and vibrancy to the in-game universe.



EDDN is not run by or affiliated with Frontier Developments.

( ).



HTTP Endpoint https://eddn.edcd.io:4430/upload/ ( ).



. ZeroMQ tcp://eddn.edcd.io:9500. .



( ):



image



, . ? , . , ( ) ! , - .





, :



image



— Microsoft Azure. - . — Azure, dotnet core/standard



--- , .



, :



1. Message Distributor



EDDN Channel . ( schemaRef , ) — . EDDN 5:





. ? , ( ), , . journal, .. ( ). , , .



, MessageDistributor EDDN . .



2. Azure Storage Queue



( Message Processor , ). , . Storage Account ( ) connection string ( , Azure AD, ). Storage Account ( — , , ).



Azure Storage Emulator Azure Storage Explorer


3. Message Processor



, , . Azure Function App.



, - ? .



, Azure Function WebJob ( , ), . , Azure Function Runtime ( , ) — — , — . , .. — .



, , (Scale Out), , , - , CPU/Memory consumption, etc.



, (in/out bindings). , , QueueTrigger — (, dotnet, ). CosmosDBTrigger . ( ). , , CosmosDB ( ) . - , , : db client ( in-built DI, ). , , queue .



QueueTrigger. , — (invisible) . , 30 ( ). 30 — DequeueCount 1. ( -> visible state on, ++DequeueCount). DequeueCount = 5 ( ), -poison. journal, 5 , journal-poison. ( - ). , . .

: WebJob AppService. — Function App. : , .



. : , , 2 . 2 , , . , (upsert — update || insert) , , ;-) 2 , , . , . , . , , , . , - (, , CosmosDB ACID compliant).



IMHO: , . . 1. , — . (, ) .



Cosmos DB Optimistic Concurrency, . . .



4. CosmosDB



. , ( ) , . , .. .



4: Signals, Systems, Stations, Bodies. , journal ( , ? -).



, journal Event, : CarrierJump, Docked, FsdJump, Location, SaaSignalsFound, Scan ( ). , — . : — . .



CosmosDB. ( , ) Request Unit per second — RU/s. .. RU/s . 2 : Provisioned throughput Serverless ( , ). , Provisioned RU/s ( ), Serverless, RU/s , , .



Provisioned RU/s. , RU/s 400, 10% 1000% (40-4000 RU/s)



RU/s Provisioned mode — 400. , , , 250 RU/s 150 , Serverless .



, ( CosmosDB). : CosmicClone. Serverless . 2 . Provisioned Mode , , .



, zero downtime "" — , MessageDistributor , . , . . .



CosmosDB

, , : EDDN , .





, React Native — dotnet, . .



: EDDNConsumer. , , .



:



  • EDDNConsumerCore — MessageDistributor. EDDN
  • EDDNModels — EDDN
  • JournalContributor — MessageProcessor journal
  • SharedLibrary — . .


EDDNConsumerCore



dotnet core 3.1 . Main DI. HostedService ConsumerService



services.AddHostedService<ConsumerService>();


nuget- ( ): NetMQ — ZeroMq Ionic.Zlib — NetMQ



StartAsync, — NetMQRuntime ClientAsync ( , ):



private async Task ClientAsync()
{
    var utf8 = new UTF8Encoding();
    using (var client = new SubscriberSocket())
    {
        client.Connect(_eddnClientSettings.ConnectionString);
        client.SubscribeToAnyTopic();
        while (true)
        {
            try
            {
                (var bytes, _) = await client.ReceiveFrameBytesAsync();
                var uncompressed = ZlibStream.UncompressBuffer(bytes);
                var result = utf8.GetString(uncompressed);
                await _messageDistributor.DistributeAsync(result);
                _logger.LogInformation(result);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error reading message queue");
            }
        }
    }
}


, ( , 1 ), _messageDistributor , , . .



MessageDistributor DistributeAsync:



public async Task DistributeAsync(string message)
{
    try
    {
        using var stringReader = new StringReader(message);
        using var jsonReader = new JsonTextReader(stringReader);
        var result = _serializer.Deserialize<Entity<BaseMessage>>(jsonReader);
        var queue = await _messageQueueFactory.GetQueueAsync(result);
        await queue.SendMessageAsync(message.Base64Encode());
    }
    catch(Exception ex)
    {
        _logger.LogError(ex, "Error distributing message");
    }
}


, , MessageQueueFactory. — . : . , schemaRef header ( test_data.json ). , , .



, ( — JSON) , . \\ , — , , null default. ( ) , POCO , , , , - . .



, , json- NewtonsoftJson ( json, ). Missing Member



_serializer.MissingMemberHandling = MissingMemberHandling.Error;
_serializer.Error += _serializer_Error;


, Application Insights, Notification Alerts . , , Header. . , .. , missing property. , , .



, header , . , ( ):



public class MessageQueueFactory : IMessageQueueFactory
{
    private readonly StorageAccount _storageOptions;
    private readonly QueueMapping _queueMapping;
    private readonly IDictionary<string, QueueClient> _queues = new Dictionary<string, QueueClient>();

    public MessageQueueFactory(
        IOptions<StorageAccount> storageOptions,
        IOptions<QueueMapping> queueMapping)
    {
        _storageOptions = storageOptions.Value;
        _queueMapping = queueMapping.Value;
    }

    public async Task<QueueClient> GetQueueAsync(Entity<BaseMessage> entity)
    {
        if (_queueMapping.TryGetValue(entity.SchemaRef, out var queueName))
        {
            if (!_queues.ContainsKey(queueName))
            {
                var client = new QueueClient(_storageOptions.StorageConnectionString, queueName);
                await client.CreateIfNotExistsAsync();
                _queues.Add(queueName, client);
            }
            return _queues[queueName];
        }
        throw new ArgumentException($"Queue {entity.SchemaRef} has not configured", nameof(entity.SchemaRef));
    }
}


, appsettings json :



{
  "QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal"
  }
}


( ):



"QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal",
    "eddn.edcd.io/schemas/blackmarket/1": "blackmarket",
    "eddn.edcd.io/schemas/commodity/3": "commodity",
    "eddn.edcd.io/schemas/outfitting/2": "outfitting",
    "eddn.edcd.io/schemas/shipyard/2": "shipyard"
  }


Dictionary<string, string> schemaRef. .



1 _queues — QueueClient, . , . , . round trip , . , , . , .



, — , — . . MessageDistributor, .



. 3 . .



EDDNModels



POCO . , . , JournalMessage:



[JsonProperty("id")]
public override string Id
{
    get => Event switch
    {
        JournalEvent.FsdJump => StarSystem,
        JournalEvent.Scan => BodyName,
        JournalEvent.Docked => StationName,
        JournalEvent.Location => BodyName,
        JournalEvent.CarrierJump => StarSystem,
        JournalEvent.SaaSignalsFound => BodyName,
        _ => $"UnknownID_{Guid.NewGuid()}"
    };
}


Id, , CosmosDB. ORM. , journal ? , . Event Id. - , - , - .



. : , , , , , , , , , \ .. ( JournalMessage). , , .. POCO — .



.



JournalContributor



"" journal — Azure Function App. Function App dotnet core class library , , . FunctionName . : — Function App ( ). - — . , . . , , ( ).



, :



public class JournalContributor
    {
        private readonly IEventTypeProcessorFactory _eventTypeProcessorFactory;

        public JournalContributor(IEventTypeProcessorFactory eventTypeProcessorFactory)
        {
            _eventTypeProcessorFactory = eventTypeProcessorFactory;
        }

        [FunctionName("ContributeJournal")]
        public async Task Run(
            [QueueTrigger("journal", Connection = "AzureWebJobsStorage")]
            Entity<JournalMessage> myQueueItem,
            ILogger log)
        {
            try
            {
                var eventProcessor = _eventTypeProcessorFactory.GetProcessor(myQueueItem.Message);
                await eventProcessor.ProcessEventAsync(myQueueItem.Message);
            }
            catch(Exception ex)
            {
                log.LogError(ex, $"Error processing queue item: {JsonConvert.SerializeObject(myQueueItem)}");
                throw;
            }
        }
    }


QueueTrigger ( ). , Event … .



, , . — , dotnet . Startup Configure, FunctionStartup (assembly):



using JournalContributor.Settings;
using Microsoft.Extensions.Configuration;
using System.IO;

[assembly: FunctionsStartup(typeof(JournalContributor.Startup))]

namespace JournalContributor
{
    public class Startup : FunctionsStartup
    {
        private IConfigurationRoot _functionConfig;
        private readonly string COSMOS_CONNECTION_STRING = Environment.GetEnvironmentVariable("CosmosDBConnectionString");
        private readonly string ENVIRONMENT = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

        public override void Configure(IFunctionsHostBuilder builder)
        {
            _functionConfig = new ConfigurationBuilder()
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, "appsettings.json"), optional: true, reloadOnChange: true)
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, $"appsettings.{ENVIRONMENT}.json"), optional: true, reloadOnChange: true)
                .Build();

            builder.Services.AddSingleton<CosmosClient>(factory => new CosmosClient(COSMOS_CONNECTION_STRING));
            builder.Services.AddSingleton<IEventTypeProcessorFactory, EventTypeProcessorFactory>();
            builder.Services.AddTransient<FsdJumpProcessor>();
            builder.Services.AddTransient<ScanProcessor>();
            builder.Services.AddTransient<DockedProcessor>();
            builder.Services.AddTransient<LocationProcessor>();
            builder.Services.AddTransient<SaaSignalsFoundProcessor>();
            builder.Services.Configure<CosmosDbSettings>(_functionConfig.GetSection("CosmosDbSettings"));
        }
    }
}


DI . , CosmosClient : (CosmosDB). ? . 2 CosmosDB — , . , .



. , , , switch :



public IEventTypeProcessor GetProcessor(JournalMessage journalMessage) => journalMessage.Event switch
        {
            JournalEvent.FsdJump => _serviceProvider.GetService<FsdJumpProcessor>(),
            JournalEvent.Scan => _serviceProvider.GetService<ScanProcessor>(),
            JournalEvent.Docked => _serviceProvider.GetService<DockedProcessor>(),
            JournalEvent.Location => _serviceProvider.GetService<LocationProcessor>(),
            JournalEvent.SaaSignalsFound => _serviceProvider.GetService<SaaSignalsFoundProcessor>(),
            JournalEvent.CarrierJump => _serviceProvider.GetService<DockedProcessor>(),
            _ => throw new ArgumentException($"Unknown Event {journalMessage.Event}", nameof(journalMessage.Event))
        };


, , . ScanProcessor, .. 2- :



public async Task ProcessEventAsync(JournalMessage message)
        {
            var existingItem = await message.CheckIfItemExists(_bodiesContainer, _options.BodiesCollection.PartitionKey);
            if (existingItem == null)
                await _bodiesContainer.UpsertItemAsync(message);
            else
            {
                //Basic scan type contains less information then others.
                //We`re skipping item upsert if remote item has scan type higher then Basic
                //We`re also skippings if remote item scan type is Detailed (as it`s a maximum info scan)
                //Will update item if both (remote and current) have scan type Basic
                //And will upsert item in case of current item have higher scan type then remote
                if ((message.ScanType == ScanType.Basic && existingItem.ScanType != ScanType.Basic) ||
                    existingItem.ScanType == ScanType.Detailed)
                    return;
                else
                    await _bodiesContainer.UpsertItemAsync(message);
            }
        }


, , ( !) , , , , . . . , . , : , Bodies Id . , .



UpSert, Insert, .. concurrency : Item with such Id already exists ( - ). , , Optimistic Concurrency . . upsert.

, ScanType. Detailed — . Basic — — . Basic — . .



. , 3 , MessageProcessor journal .





, . , Azure WebJob EDDN , Storage Account , Azure Function App Cosmos DB . Application Insights (, -, , ). , .



image



continuous, .. .



![image](img src="https://habrastorage.org/webt/i_/5u/zq/i_5uzqxkanblmotxacqhtwnp_wq.png)



, , schemaRef . , .



image



Stream-log : , , … .



image



. , , .



image



RU/s . — Provisioned Mode 400 RU/s. , . ~105RU/s.



image



. 145 . EDSM, .



image



. Check.



. . .





. . - .



45 . , MSDN , 45 . . , . . 45 — 1.5 . , — .



— . - — , - — "" . .



image



. ~10 . appinsjournalweprivate . Function App. — , . \. :



image



( ), host.json JournalContributor:



{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    },
    "logLevel": {
      "default": "Information",
      "Host": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }
  }
}


:



image



, ( 21 ):



image



( 193 ). - / . 21 1.75 . — appinsweprivate — WebJob. 0.98 0.19 21 .



cdbweprivate — CosmosDB Provisioned Mode 400 RU/s. . cdbslweprivate — Serverless mode CosmosDB. 4.47 .



— laweprivate. Log Analytics Workspace, , , .. CosmosDB. , , , . .



, 175,8 . , .



, . -. appinsights — 3 . , . , % , . - , , 3 — ( ).



-. CosmosDB RU/s ( , ) RU/s (Pricing), :



image



EDDN. , , Elite: Dangerous. - " " — . , , DLC Odyssey , . .



, Function App Storage Account, , .



, . — .



. .





.



" " " ", . - , . =) , -, ...



— . , - , 145 , EDSM 57 . - - , .



, , , . . " " — .



image



, — . ? . , , EDSM . , JSON — . — ( ). RU/s , , . .



CosmosDB — Jupyter Notebooks. C#, Python, - . , .



, , . ( , Sol [0,0,0]:



image



? :



image



5 :



image



EDSM, - , . , , — - . , , .



CosmosDB, Azure Function Change Feed ( ), .



— CosmosDB .



Pour ceux qui ont enduré jusqu'au bout et qui sont toujours intéressés par où chercher des planètes ressemblant à la Terre dans l'élite, voici votre réponse (les résultats sont basés sur des données obtenues à partir d'une décharge EDSM complète il y a environ six mois):



image



Les types F, K, G, A et, pour une raison quelconque, les neutrons constituent les 5 principaux types d'étoiles à partir desquelles les pilotes ont trouvé des planètes semblables à la Terre. Eh bien, ce sont les résultats ¯ \ _ (ツ) / ¯



Bien. C'est tout.



Volez en toute sécurité, cmdr!



More articles:


All Articles