Category: Dev

My latest project has gone online

Posted in Dev, SecurityLeave a comment

For some days now, a project I’ve been working on for some time has been online: it is a solution for monitoring the Internet. Right now the main focus is on the search for phishing kits and webshell. As it is designed, the solution can perform various other tasks but given the consumption of resources that this type of work entails (and that the resources available, already like this, are almost at the limit) for now I am limiting its functionality.

It is a project that I carry out as a hobby so I work on it in the spare time, but lately it has been giving me some satisfaction!

The thing I like most about this project is that it presents a significant amount of difficulties that allow me to always learn new things on different fronts: design, programming, databases, systems, networks and of course, security.

If you are interested in the result of this project you can find the results on my Twitter account (which has now become unusable given the amount of messages that the platform produces …).

The solution consists of several subsystems and some of these I have decided to release on Github. The first project that I released is Argilla, it is still in beta and has some bugs as well as being poor in functionality, but it is light and does what I need right now. When I have time I will evolve it to make it more usable in other contexts.

Currently 4000000 of websites are analyzed every day, for each site the content is analyzed and a value indicating the level of risk is calculated. Sites with higher risk values ​​are published on Twitter as “Threat” and “Possible threat”. Before being published on Twitter they are reported to Netcrat through the appropriate API. The publication on Twitter includes, in addition to the link, some hashtags (mainly #phishing and #opendir) and, since the last release, the registrar who registered the domain is tagged.

The application components are developed in .NET, Core and Framework, the operating systems are Windows 2019 (for the database) and Linux for the agents. The database is SQL Server 2019. The databases are 4 for a total of about 300GB and grow by about 6GB per day.

Right now I am evaluating the possibility of adopting Tensorflow for some works but I am having some difficulty in creating useful datasets. If anyone has skills and wants to work on this project, any help is welcome!

Argilla

Posted in DevLeave a comment

Argilla is my new project available on Github.

Argilla is a simple distributed bus for the integration between microservices. The adoption of Argilla allows the elimination of microservice endpoint configuration so that they can be managed more easily.

At present the project is functional even if it is not performing and unstable. Your help could be very useful to improve it 😉

The idea of this project starts from an internal need of my cyber security project developed in .NET Core on Linux. I needed to enable various services to communicate without having to manage a catalog and even worse a distributed configuration. I also had the need to have callbacks from calls that can last for hours.

Creating an Argilla enabled microservice is very simple, just add a reference, a few lines of code and your microservice are ready.

The architecture of Argilla is very simple, there is a Resolver server and its endpoint is configured in the various microservices. Argilla automatically publishes the catalog of services endpoint so that clients can consume the services without previously knowing their locations. If multiple services implement the same service, the client takes turns invoking the services to distribute the load. The client then takes care not to invoke the offline services and turn the requests to those available.

Wait for an event in a WaterfallDialog

Posted in DevLeave a comment

Working on a Skill, I needed to block the execution of a WaterfallDialog while waiting for a specific event to arrive.
The solution is simple and based on the use of EventPrompt. Below I show how to use it.

Register EventPrompt in the constructor

AddDialog(new WaterfallDialog(nameof(WaterfallDialog), new WaterfallStep[]
{
	HelloAsync,
	WaitForEventAsync,
	DoneAsync
}));

AddDialog(new EventPrompt(nameof(EventPrompt), "SampleEvent", Validate));

InitialDialogId = nameof(WaterfallDialog);

Implement the three dialog steps

private async Task<DialogTurnResult> HelloAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
	await stepContext.Context.SendActivityAsync(MessageFactory.Text("Hello :)"), cancellationToken);

	return await stepContext.NextAsync(cancellationToken);
}

private async Task<DialogTurnResult> WaitForEventAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
	return await stepContext.PromptAsync(nameof(EventPrompt), new PromptOptions { Prompt = MessageFactory.Text("Waiting for the event") });
}

private async Task<DialogTurnResult> DoneAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
	await stepContext.Context.SendActivityAsync(MessageFactory.Text("After I receive the event."), cancellationToken);

	return await stepContext.NextAsync(cancellationToken);
}

Implement the event validator

async Task<bool> Validate(PromptValidatorContext<Activity> promptContext, CancellationToken cancellationToken)
{
	var activity = promptContext.Recognized.Value;
	if (activity.Type == ActivityTypes.Event)
	{
		try
		{
			string eventValue = (string)activity.Value;
			
			if(eventValue == "OK")
			{
				return true;
			}
		}
		catch { }
	}

	return false;
}

Share data from Virtual Assistant to Skill

Posted in DevLeave a comment

For some weeks I have been working on a project based on Bot Framework. Things were going well until I found myself in the position of having to share information between a Virtual Assistant and a Skill … Given that it gave me problems and that my requests made on Stackoverflow did not produce answers, I share the solution here.

This post expects the user to be clear what Virtual Assistant and Skill are and has already installed the prerequisites as described here:
https://microsoft.github.io/botframework-solutions/tutorials/csharp/create-assistant/1_intro/

Create a Virtual Assistant using the Virtual Assistant Template and name it SampleVirtualAssistant

Add to the solution SampleVirtualAssistant a new project (the Skill) using the Skill Template and name it SampleSkill

Rebuild the solution.

In the following example I’ll show the commands as execute on my system, where my sources path is C:\Users\emili\source\lab, be aware to replace it with your correct path!

Open PowerShell Core to execute the following commands.

Deploy to Azure the required resources.

cd c:\users\emili\source\lab\SampleVirtualAssistant\SampleVirtualAssistant\SampleVirtualAssistant
.\Deployment\Scripts\deploy.ps1
cd C:\Users\emili\source\lab\SampleVirtualAssistant\SampleSkill\SampleSkill
.\Deployment\Scripts\deploy.ps1

Now we go to work on the code.

In the project SampleSkill, edit the file manifestTemplate.json to add the required slot.

cd C:\users\emili\source\lab\SampleVirtualAssistant\SampleVirtualAssistant\SampleVirtualAssistant
botskills connect --botName SampleVirtualAssistant --remoteManifest "http://<your resource>.azurewebsites.net/api/skill/manifest" --luisFolder C:\Users\emili\source\lab\SampleVirtualAssistant\SampleSkill\SampleSkill\Deployment\Resources\LU\en\ --cs

In the project SampleVirtualAssistant edit the file skills.json to add the required slot.

In the project SampleVirtualAssistant edit the file MainDialog.cs to update the class Events

In the same file, in the method OnEventAsync add the code that handle the event to set the UserEmail

In the project SampleSkill, in the file MainDialog.cs in the RouteAsync method insert this code after the call to PopulateSateFromSemanticAcion

Rebuild all.

Publish the skill and the VA.

Start the chat and test it 🙂

Azure Key Vault – Basic

Posted in DevLeave a comment

Over time, I have used Key Vault several times. I think it’s a great solution and I clearly recommend its use to everyone!

Since every time I talk about it with something, after the enthusiasm I am asked how to use it, I have prepared a simple project that illustrates the basics of its use.

Use this PowerShell commands to create the Vault:

# Using this command you can see all the location available
az account list-locations

az login

# The following command a new resource group. If you already have one you can use it. 
# Choose your location
az group create -n "resource-group-name" -l "North Europe"

az provider register -n Microsoft.KeyVault

# Create the Key Vault. This call return the URL of the key vault.
az keyvault create --name "keyvault-name" --resource-group "resource-group-name" --location "North Europe"

# Add some secrets to the vault
az keyvault secret set --vault-name "keyvault-name" --name "secret-1" --value "test 1"
az keyvault secret set --vault-name "keyvault-name" --name "secret-2" --value "test 2"
az keyvault secret set --vault-name "keyvault-name" --name "secret-3" --value "test 3"

# List all the secrets in the specified vault
az keyvault secret list --vault-name "keyvault-name"

# Create an app. this call return the appId and secret to use in the app.config
az ad sp create-for-rbac -n "app-name" --skip-assignment

# Trust the key vault to be accessed with the app credentials
az keyvault set-policy --name "keyvault-name" --spn <use the appId previously created> --secret-permissions get list set

In this example, credentials are used to log in to the vault, I recommend, if you have an application in Azure, to set up an identity to log in to the vault.

Here you can find a .NET example project:
https://github.com/ecarlesi/azure-keyvault-sampleclient