MR70

Connecting Microsoft Copilot Studio to an AWS Bedrock AgentCore Gateway

Tue, 26 May 2026 00:00:00 +0000

Overview

Microsoft Copilot Studio supports the Model Context Protocol (MCP) to invoke external tools. AWS Bedrock AgentCore Gateway exposes MCP servers behind a JWT authentication layer. Connecting the two requires:

A Cognito User Pool with an app client configured for the Authorization Code flow
An AgentCore Gateway with a CUSTOM_JWT authorizer that accepts the Copilot Studio client
A Gateway Target pointing to the MCP runtime

The resulting architecture:

Copilot Studio → (OAuth2 Authorization Code) → Cognito → JWT
     ↓
AgentCore Gateway (validates JWT) → MCP Runtime → Tool

Prerequisites

An AgentCore Gateway with a target configured to an MCP runtime
An Amazon Cognito User Pool with an active hosted UI domain
Admin access to the Copilot Studio portal

Step 1 — Configure the Cognito app client

The app client must support the code (Authorization Code) flow with a client secret:

aws cognito-idp create-user-pool-client \
  --region eu-west-1 \
  --user-pool-id  \
  --client-name "copilot-studio-mcp" \
  --generate-secret \
  --allowed-o-auth-flows code \
  --allowed-o-auth-scopes openid email profile mcp/invoke \
  --allowed-o-auth-flows-user-pool-client \
  --supported-identity-providers COGNITO \
  --callback-urls "https://placeholder.example.com"

The custom scope mcp/invoke must exist on a resource server in the User Pool. Create it if it doesn’t exist:

aws cognito-idp create-resource-server \
  --region eu-west-1 \
  --user-pool-id  \
  --identifier mcp \
  --name "MCP Server Access" \
  --scopes '[{"ScopeName":"invoke","ScopeDescription":"Invoke MCP server tools"}]'

Step 2 — Authorize the client on the Gateway

The Gateway must accept the new app client’s client_id in its allowedClients list:

aws bedrock-agentcore-control update-gateway \
  --region eu-west-1 \
  --gateway-id  \
  --name  \
  --role-arn  \
  --authorizer-type CUSTOM_JWT \
  --protocol-configuration '{
    "mcp": {
      "supportedVersions": ["2025-03-26"],
      "streamingConfiguration": {"enableResponseStreaming": false}
    }
  }' \
  --authorizer-configuration '{
    "customJWTAuthorizer": {
      "discoveryUrl": "https://cognito-idp.eu-west-1.amazonaws.com//.well-known/openid-configuration",
      "allowedClients": ["", ""],
      "allowedScopes": ["mcp/invoke", "profile", "openid", "email"]
    }
  }'

Step 3 — Configure Copilot Studio

In Copilot Studio: Agent → Tools → Add a tool → Add a Model Context Protocol server.

Select Manual as the OAuth type and fill in:

Field	Value
Client ID	The `client_id` created in Step 1
Client secret	The client’s `client_secret`
Authorization URL	`https://.auth.eu-west-1.amazoncognito.com/oauth2/authorize`
Token URL	`https://.auth.eu-west-1.amazoncognito.com/oauth2/token`
Refresh URL	(same as Token URL)
Scopes	`openid email profile mcp/invoke`

After clicking Create, Copilot Studio generates a Redirect URL.

Step 4 — Register the Redirect URL in Cognito

Add the Copilot Studio redirect URL to the client’s allowed callbacks:

aws cognito-idp update-user-pool-client \
  --region eu-west-1 \
  --user-pool-id  \
  --client-id  \
  --callback-urls "https://global.consent.azure-apim.net/redirect/" \
  --supported-identity-providers COGNITO \
  --allowed-o-auth-flows code \
  --allowed-o-auth-scopes openid email profile mcp/invoke \
  --allowed-o-auth-flows-user-pool-client

Step 5 — Test the connection

Back in Copilot Studio, click Next → Create a new connection and authenticate with a Cognito user. If everything is configured correctly, Copilot Studio obtains an access token and starts invoking MCP tools.

Troubleshooting

Error	Cause	Fix
`insufficient_scope`	The `client_id` is not in the Gateway’s `allowedClients`	Add it via `update-gateway` (Step 2)
`invalid_scope`	The scope doesn’t exist on the resource server or isn’t enabled on the client	Verify resource server and `AllowedOAuthScopes`
`redirect_uri mismatch`	The Copilot redirect URL is not in the client’s callbacks	Run Step 4
`unsupported_grant_type`	The client doesn’t have the `code` flow enabled	Add `code` to `AllowedOAuthFlows`
`Failed to fetch outbound oauth token`	The target’s credential provider requests scopes invalid for `client_credentials`	Use only custom scopes (e.g. `mcp/invoke`), not OIDC scopes

MCP protocol version

Copilot Studio currently supports MCP protocol version 2025-03-26. Make sure the Gateway is configured with this version in supportedVersions, otherwise the connection will fail silently.

Installing Ubuntu (and valetudo app) on my old Xiaomi Vacuum cleaner gen 1

Wed, 06 Aug 2025 00:00:00 +0000

Installing Ubuntu (and valetudo app) on my Xiaomi Vacuum cleaner gen 1

I have installed ubuntu on my old Xiaomi Vacuum cleaner gen 1 (< year 2000) using OTA update. Now it cleans and speaks Italian funny messages with the voice of my children.

I just followed the suggestions from

official valetudo website (https://builder.dontvacuum.me/index_xiaomi.html) and
https://www.ajfriesen.com/cloud-free-robot-vacuum/
https://github.com/dgiese/dustcloud/tree/master/devices/xiaomi.vacuum/audio_generator (/opt/rockrobo/resources/sounds/PRT)

Before I had to hard reset it (pressing “reset button” with both buttons, after some seconds I released “reset button” and kept pressed the 2 buttons until I listened a chinese voice)

System info

Linux rockrobo 3.4.39 #1 SMP PREEMPT Sat Jan 9 09:34:52 UTC 2021 armv7l armv7l armv7l GNU/Linux

Disks

Filesystem Size Used Avail Use% Mounted on /dev/root 494M 390M 79M 84% / devtmpfs 202M 4.0K 202M 1% /dev none 4.0K 0 4.0K 0% /sys/fs/cgroup tmpfs 30M 1.7M 29M 6% /tmp none 50M 236K 50M 1% /run none 5.0M 0 5.0M 0% /run/lock tmpfs 100M 1.7M 99M 2% /run/shm none 100M 0 100M 0% /run/user /dev/mmcblk0p11 16M 1.6M 14M 11% /mnt/reserve /dev/mmcblk0p6 16M 1.2M 14M 8% /mnt/default /dev/mmcblk0p10 494M 261M 208M 56% /mnt/updbuf /dev/mmcblk0p1 1.6G 176M 1.4G 12% /mnt/data

CPUs

Processor : ARMv7 Processor rev 5 (v7l) processor : 0 BogoMIPS : 3428.56

processor : 1 BogoMIPS : 3428.56

processor : 2 BogoMIPS : 3428.56

processor : 3 BogoMIPS : 3428.56

Features : swp half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpv4 idiva idivt CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x0 CPU part : 0xc07 CPU revision : 5

Hardware : sun8i Revision : 0000

Memory

MemTotal: 510548 kB

msad - a cli for microsoft active directory

Sun, 20 Apr 2025 00:00:00 +0000

msAD

msad is a library and command line tool for working with an Active Directory / LDAP server from Unix, Linux and MacOs systems.

It supports authentication with user/pwd and kerberos

It supports paginations: it can retreive more than 2000 objects (a limit of AD)

Features:

search objects (users, groups, computers,..)
search (recursively) group memberships and all user’s groups
add/remove members to/from AD groups using DN or sAMaccoutName
change AD passwords

Usage

msad search “(samaccountname=matteo)” –out-format=json # show all attributes

msad search “(cn=redaelli*)” –attributes mail –attributes samaccountname –out-format=json

msad group-members qlik_analyzer_users –nested

msad group-add-member qlik_analyzer_users matteo

msad group-remove-member qlik_analyzer_users matteo

msad user-groups matteo –nested

Camel K: howto deploy Apache Camel integrations in Kubernetes

Sat, 22 Mar 2025 00:00:00 +0000

Abstract

In this article I’ll show how to

create a Apache Camel integration (read data from a database Aws Redshift, processing each row calling a rest service and finally saving the output to aws S3 files).
deploy the integration to a Kubernetes (k8s) cluster using the command line tool Kamel. I’ll use Minikube as a local Kubernetes cluster. Read the official documentation for installing it.

Setup kamel

As suggested by Camel-k installation

minikube start
minikube addons enable registry

kubectl create ns camel-k
kubectl config set-context --current --namespace=camel-k

kubectl apply -k github.com/apache/camel-k/install/overlays/kubernetes/descoped?ref=v2.6.0 --server-side

Create a file itp.yaml like

apiVersion: camel.apache.org/v1
kind: IntegrationPlatform
metadata:
  labels:
    app: camel-k
  name: camel-k
  namespace: camel-k
spec:
  build:
    registry:
      address: registry.io
      organization: camel-k
      insecure: true

And update the “address” with the ip shown by

kubectl -n kube-system get service registry -o jsonpath='{.spec.clusterIP}'

Then run

kubectl apply -f itp.yaml -n camel-k
kubectl wait --for jsonpath='{.status.phase}'=Ready IntegrationPlatform camel-k --timeout 30s

Creating an integration with Camel

File dbToRestToS3.yaml

- from:
    uri: "direct:process_record"
    steps:
      - log: "${body}"
      - log: "${body['col1']}"
      - setHeader:
          name: "S3Filename"
          expression:
            simple: "testCanBeDelete/${body['col1']}"
      - setHeader:
          name: "CamelHttpUri"
          expression:
            simple: "https://api.restful-api.dev/objects?id=${body['col1']}"
      - setHeader:
          name: "CamelHttpMethod"
          constant: "GET"
      - to: "http://RESTSERVICE"
      - log: "${body}"
      - log: "${in.headers}"
      - to:
          uri: "kamelet:aws-s3-sink/myconfig"
          parameters:
            keyName: "${in.headers['S3Filename']}"

- from:
    uri: "direct:start"
    steps:
      - setBody:
          constant: "{}"
      - to:
          uri: "kamelet:aws-redshift-sink"
          parameters:
            query: select '3' as col1 union select '4' as col1
      - split:
          expression:
            simple: "${body}"
          steps:
            - to:
                uri: "direct:process_record"
      
- from:
    uri: "timer:tick?period=30000"
    steps:
      - to:
          uri: "direct:start"

# - from:
#     uri: "cron:tab?schedule=*+*+*+*+?"
#     steps:
#       - to: "direct:start"

File redshift-sink.properties:

camel.kamelet.aws-redshift-sink.serverPort: 5439
camel.kamelet.aws-redshift-sink.username=myuser
camel.kamelet.aws-redshift-sink.password=myprd
camel.kamelet.aws-redshift-sink.query=select user
camel.kamelet.aws-redshift-sink.databaseName=mydb

File aws-sink.properties:

camel.kamelet.aws-s3-sink.myconfig.bucketNameOrArn=mybucket
camel.kamelet.aws-s3-sink.myconfig.region=eu-west-1
camel.kamelet.aws-s3-sink.myconfig.keyName=testCanBeDelete/replaceme
camel.kamelet.aws-s3-sink.myconfig.accessKey=myaccessKey
camel.kamelet.aws-s3-sink.myconfig.secretKey=mysecretKey

Deployment

kamel run dbToRestToS3.yaml --property file:redshift-sink.properties --property file:aws-sink.properties --dev

Or using secrets

kubectl create secret generic aws-sink-secret --from-file=aws-sink.properties
kubectl create secret generic redshift-sink-secret --from-file=redshift-sink.properties
kamel run --dev --config secret:redshift-sink-secret --config secret:aws-sink-secret dbToRestToS3.yaml

Logging

In production you need to omit the option “–dev” and see logs with

kubectl get pods -n 
kubectl logs camel-k-operator-XXXX
kamel get
kamel logs db-to-rest-to-s3

References

Mr Bates vs The Post Office

Fri, 14 Mar 2025 00:00:00 +0000

Below my #prolog solution for Mr Bates vs The Post Office.

The Post Office in DS City has come up with a new measure to limit packages that can be sent under a new flat rate, regardless of weight or volume. The “strap measure” of a box is the length of the longest side plus two times the sum of the two shorter sides. For the new flat rate for sending a packet, the strap measure may not exceed 100 inches.

Your e-commerce client Mr. Bates sells large comforters of 9,000 cubic inches in volume and wants to make use of this new flat rate if possible. What should the measurements of the packaging box be to maximize the volume of the package? Can you design a box that will be large enough to hold these comforters and that can be sent at the flat rate price?

swipl dm-commumity-2025-02.pl
?- findall([X,Y,Z], sol(X,Y,Z), Sol).
   Sol = [[30, 20, 15], [40, 15, 15]].

Below the Prolog code

:-use_module(library(clpfd)).

sol(X,Y,Z):-
    X #>= 1,
    Y #>= 1,
    Z #>= 1,
    9000 #= X * Y * Z,
    X #>= Y,
    Y #>= Z,
    100 #>= X + 2 * (Y + Z),
    label([X,Y,Z]).

Setting up a workstation/server and managing dotfiles centrally using Ansible and GNU Stow

Mon, 09 Sep 2024 00:00:00 +0000

About

In the world of software development, maintaining a consistent and efficient development environment across multiple machines can be challenging. This blog post explores how to leverage two powerful tools, Ansible and GNU Stow, to automate the setup of your development environment and manage your dotfiles effectively.

The Power of Automation with Ansible

Ansible is an open-source automation tool that excels in configuration management, application deployment, and task automation. Its key advantages include:

Agentless architecture: Ansible doesn’t require any agents to be installed on managed nodes, making it lightweight and easy to set up.
YAML-based playbooks: Ansible uses human-readable YAML files to describe automation jobs, making it easy to write and understand.
Idempotency: Ansible ensures that running the same playbook multiple times produces the same result, making it safe to re-run automation scripts.
Extensibility: With a vast collection of modules, Ansible can automate almost any IT task.

Simplifying Dotfile Management with GNU Stow

GNU Stow is a symlink farm manager that takes a different approach to managing dotfiles:

Symlink-based: Stow creates symlinks from your home directory to your actual dotfiles, keeping them organized in a separate directory.
Package-oriented: It allows you to group related dotfiles into “packages,” making it easy to manage configurations for different tools.
Non-intrusive: Stow doesn’t overwrite existing files, ensuring the safety of your current configurations.
Version control friendly: By keeping your dotfiles in a separate directory, it’s easy to version control them with Git or other VCS tools.

Putting It All Together

mkdir $HOME/ansible_dotfile # should be in the $HOME
mkdir $HOME/ansible_dotfile/dotfiles
mkdir $HOME/ansible_dotfile/dotfiles_work

Now put your configuration files (like .bash_profile, .config/emacs/init.el, .ssh/config in the “dofiles” subfolder (like $HOME/ansible_dotfile/dotfiles/.config/emacs/init.el)

Let’s look at how we can combine Ansible and GNU Stow to set up a development environment and manage dotfiles. Here’s a breakdown of the sample Ansible playbook bootstrap.yml:

## Run with ansible-playbook bootstrap.yml
- name: Bootstrap environment for Matteo laptop
  hosts: localhost
  tasks:
    - name: Update homebrew and upgrade all packages
      community.general.homebrew:
        update_homebrew: true
        upgrade_all: true

    - name: Install packages with brew
      community.general.homebrew:
        name:
          - awscli
          - dbeaver-community
          - devtunnel
          - emacs
          - firefox
          - google-chrome
        state: present
      when: ansible_distribution == "MacOSX"
      
    - name: Install languages packages with brew
      community.general.homebrew:
        name:
          - clojure
          - erlang
          - jq
          - python@3.12
          - swi-prolog
        state: present
      when: ansible_distribution == "MacOSX"

    - name: Run stow
      shell: "stow dotfiles  --verbose=2 --no-folding -R"
      register: result
      changed_when: 'result.stderr is search("LINK: ")'  

    - name: Run stow for my job config files
      shell: "stow dotfiles_work  --verbose=2 --no-folding -R"
      register: result
      changed_when: 'result.stderr is search("LINK: ")'

    - name: Steampipe plugins
      shell: "steampipe plugin install jira whois aws ldap net rss"
      register: result
      changed_when: 'result.stderr is search("Installed plugin")'

    - name: Steampipe plugins
      shell: "steampipe plugin update --all"
      register: result

This playbook does the following:

Updates Homebrew and all installed packages.
Installs a set of common development tools and applications using Homebrew.
Installs programming languages and related tools.
Uses GNU Stow to manage dotfiles for both personal and work environments.
Installs and updates Steampipe plugins for various services.

Benefits of This Approach

Reproducibility: Your entire development environment can be recreated with a single command.
Version Control: Both your Ansible playbook and dotfiles can be version-controlled, allowing for easy tracking of changes and rollbacks if needed.
Flexibility: The playbook can be easily modified to add or remove packages as your needs change.
Cross-platform: With proper conditionals, you can use the same playbook across different operating systems.
Separation of Concerns: Personal and work-related dotfiles are managed separately, allowing for easy context switching.

Conclusion

By combining Ansible’s powerful automation capabilities with GNU Stow’s elegant dotfile management, you can create a robust, version-controlled, and easily reproducible development environment. This approach not only saves time in setting up new machines but also ensures consistency across all your development environments.

Remember to keep your Ansible playbook and dotfiles in a version control system like Git, and consider hosting them on a platform like GitHub for easy access across different machines. Happy coding!


This blog post provides an overview of using Ansible and GNU Stow for development environment setup and dotfile management, explains the advantages of both tools, and breaks down the sample configuration file you provided. It should give readers a good understanding of how to implement this approach in their own workflows.

Smart investment Problem with Prolog

Sun, 14 Jul 2024 00:00:00 +0000

Below my #prolog solution for Smart investment problem.

It is a sample of Linear Programming using Prolog.

A client of an investment firm has $10000 available for investment. He has instructed that his money be invested in particular stocks, so that no more than $5000 is invested in any one stock but at least $1000 be invested in each stock. He has further instructed the firm to use its current data and invest in the manner that maximizes his overall gain during a one-year period.

?- sol(S), variable_value(S, x1, QuantityABC), variable_value(S, x2, QuantityXYZ), variable_value(S, x3, QuantityTTT), variable_value(S, x4, QuantityLMN).

QuantityABC = 120,
QuantityXYZ = 20,
QuantityTTT = 10,
QuantityLMN = 200.

Below the Prolog code

:- use_module(library(simplex)).

sol(S) :-
    gen_state(S0),
    my_constraints(S0, S1),
    maximize([35*x1, 60*x2, 125*x3, 40*x4], S1, S).

my_constraints -->
    constraint([25*x1] >= 1000),
    constraint([25*x1] =< 5000),
    constraint([50*x2] >= 1000),
    constraint([50*x2] =< 5000),
    constraint([100*x3] >= 1000),
    constraint([100*x3] =< 5000),
    constraint([25*x4] >= 1000),
    constraint([25*x4] =< 5000),
    constraint([25*x1, 50*x2, 100*x3, 25*x4] =< 10000),
    constraint([x1] >= 0),
    constraint([x2] >= 0),
    constraint([x3] >= 0),
    constraint([x4] >= 0).

DevOps Tableau Cloud - managing tableau cloud resources with OpenTofu / Terraform

Wed, 26 Jun 2024 09:14:35 +0000

Abstract

OpenTofu (a Terraform fork, community-driven and managed by the Linux Foundation) is an open-source infrastructure as code (IaC) tool that allows you to define, provision, and manage infrastructure as code. Terraform can be used to manage a wide range of infrastructure resources, including Tableau Cloud resources.

Here are some of the benefits of using OpenTofu/Terraform to manage Tableau Cloud resources:

Consistency: Terraform allows you to define your infrastructure in a consistent and repeatable way. This can help to reduce errors and improve the overall quality of your infrastructure.

Automation: Terraform can be used to automate the deployment and management of your Tableau Cloud resources. This can save you time and effort, and it can also help to improve the reliability of your infrastructure.

Version control: Terraform allows you to version control your infrastructure configuration. This means that you can track changes to your infrastructure over time and easily roll back to previous versions if necessary.

To get started with OpenTofu/Terraform, you will need to install the OpenTofu/Terraform CLI tool. Once you have installed the OpenTofu/Terraform CLI, you can create a new Terraform configuration file. Terraform configuration files are written in a language called HashiCorp Configuration Language (HCL).

Usage

Below I’ll show how to create some Tableau groups, users and projects.

Create a folder and inside the following files:

File versions.tf

terraform {
    required_providers {
	tableau = {
	    source = "GtheSheep/tableau"
	    version = "0.0.17"
	}
    }
}

File providers.tf

provider "tableau" {
    server_url = "https://eu-west-1a.online.tableau.com"
    server_version = "3.13"
    site           = "redaelli"
}

File tableau_groups.tf

resource "tableau_group" "group_developer" {
    provider = tableau
    name     = "developer"
    minimum_site_role = "Creator"
}

File tableau_users.tf

resource "tableau_user" "matteo-redaelli" {
    provider = tableau
    auth_setting = "ServerDefault"
    name  = "matteo@redaelli.org"
    email = "matteo@redaelli.org"
    full_name = "matteo@redaelli.org"
    site_role = "Viewer"
}

resource "tableau_group_user" "group_developer_user_matteo-redaelli" {
    provider = tableau
    group_id = "${resource.tableau_group.group_developer.id}"
    user_id  = "${resource.tableau_user.user_matteo-redaelli.id}"
}

File tableau_projects.tf

resource "tableau_project" "project_crm" {
    provider = tableau
    name     = ""
    description = "CRM projects"
    parent_project_id = ""
    content_permissions = "ManagedByOwner"
}

resource "tableau_project" "project_crm_sales" {
    provider = tableau
    name     = "CRM Sales"
    description = "CRM Sales"
    parent_project_id = "${resource.tableau_project.project_crm.id}"
    content_permissions = "ManagedByOwner"
}

Now you can create your resources with


tofu init
tofu plan -out myplan
tofu apply myplan

References:

OpenTofu is a fork of Terraform under the Linux Foundation unbrella.
GtheSheep/terraform-provider-tableau
Tableau

Stable Marriage Problem with Prolog

Fri, 07 Jun 2024 00:00:00 +0000

My #prolog solution for Stable Marriage Problem proposed by dmcommunity.org challenge Jun 2024.

Given n men and n women, where each person has ranked all members of the opposite sex in order of preference, marry the men and women together such that there are no two people of opposite sex who would both rather have each other than their current partners. When there are no such pairs of people, the set of marriages is deemed stable.

Algorithms for finding solutions to the stable marriage problem have applications in a variety of real-world situations, perhaps the best known of these being in the assignment of graduating medical students to their first hospital appointments

?- findall(M, (sol(S), marriages(S, M)), L), maplist(writeln, L).

[Adam-Claire,Bob-Barbara,Charlie-Alice,Dave-Elsie,Edgar-Doris]
[Adam-Claire,Bob-Doris,Charlie-Alice,Dave-Elsie,Edgar-Barbara]
[Adam-Doris,Bob-Elsie,Charlie-Alice,Dave-Claire,Edgar-Barbara]
[Adam-Elsie,Bob-Claire,Charlie-Alice,Dave-Doris,Edgar-Barbara]
[Adam-Elsie,Bob-Claire,Charlie-Doris,Dave-Barbara,Edgar-Alice]

Below the code

:-use_module(library(clpfd)).

dim(5).

husbands_ranks([[5,1,2,4,3],
		[4,1,3,2,5],
		[5,3,2,4,1],
		[1,5,4,3,2],
		[4,3,2,1,5]
	       ]).

wifes_ranks([[5,1,2,4,3],
	     [4,1,3,2,5],
	     [5,3,2,4,1],
	     [1,5,4,3,2],
	     [4,3,2,1,5]
	    ]).

rank(Person1, Person2, Ranks, Rank):-
    nth1(Person1, Ranks, PersonRanks),
    nth1(Person2, PersonRanks, Rank).

/*
Wifes is a list of integer
First element is the wife of men 1
Second element is the wife of men 2
...
*/
sol(Wifes):-
    dim(N),
    husbands_ranks(HusbandRanks),
    wifes_ranks(WifeRanks),
    %% Wifes is a list of N elements
    length(Wifes, N),
    %% Each element is a wige (1=Alice)
    Wifes ins 1..N,
    all_distinct(Wifes),
    label(Wifes),
    stable_marriages(Wifes, HusbandRanks, WifeRanks).

instable_marriages(Wifes, HusbandRanks, WifeRanks, [Husband1-Wife1, Husband2-Wife2]):-
    %% Husband1 is the index of an element of the list
    %% Wife1 is a element of teh list of 
    nth1(Husband1, Wifes, Wife1),
    rank(Husband1, Wife1, HusbandRanks, MRank11),
    nth1(Husband2, Wifes, Wife2),
    Husband1 #\= Husband2,
    Wife1 #\= Wife2,
    rank(Wife2, Husband2, WifeRanks, FRank22),
    rank(Husband1, Wife2, HusbandRanks, MRank12),   
    rank(Wife2, Husband1, WifeRanks, FRank21),
    MRank12 #> MRank11,
    FRank21 #> FRank22.

stable_marriages(Wifes, HusbandsRanks, WifesRanks):-
    instable_marriages(Wifes, HusbandsRanks, WifesRanks, _) -> false ; true.

%% needed to convert numbers to first names

husbands_names(['Adam','Bob','Charlie','Dave','Edgar']).
wifes_names(['Alice','Barbara','Claire','Doris','Elsie']).

wife_index_name(Index, Name):-
    wifes_names(WifesNames),
    nth1(Index, WifesNames, Name).

wifes_index_name([], []).
wifes_index_name([Index|Indexes], [Name|Names]):-
    wife_index_name(Index, Name),
    wifes_index_name(Indexes, Names).
    
marriages(Wifes, Marriagies):-
    husbands_names(HusbandsNames), 
    wifes_index_name(Wifes, WifesNames),
    pairs_keys_values(Marriagies, HusbandsNames, WifesNames).

DM Community - challenge March 2024 - generative ai

Thu, 28 Mar 2024 00:00:00 +0000

Usually I solve DM Community challenges using the Prolog language. Regarding https://dmcommunity.org/challenge/challenge-march-2024/ I used the generative AI.

Below the prompt I used but the results are not always good! What’s wrong? How could I change my prompt in order to get better results?

Hello. 
Act as developer. 
Given a list of "employees" in the following json format: '{
  "company" : {
    "companyName" : "ABC",
    "employees" : [ {
      "name" : "Robinson",
      "age" : 25,
      "gender" : "Female",
      "maritalStatus" : "Married",
      "minor" : false,
      "locations" : [ {
        "id" : "RobinsonLoc1",
        "street" : "Main Str",
        "zipCode" : "08831",
        "state" : "NJ"
      }, {
        "id" : "RobinsonLoc2",
        "street" : "Ocean Drive",
        "zipCode" : "33019",
        "state" : "FL"
      } ],
      "children" : 2,
      "salary" : 220000.0
    }, {
      "name" : "Warner",
      "age" : 45,
      "gender" : "Male",
      "maritalStatus" : "Married",
      "minor" : false,
      "locations" : [ {
        "id" : "WarnerLoc",
        "street" : "Maple Street",
        "zipCode" : "08817",
        "state" : "NJ"
      } ],
      "children" : 0,
      "salary" : 150000.0
    }, {
      "name" : "Stevens",
      "age" : 24,
      "gender" : "Male",
      "maritalStatus" : "Single",
      "minor" : false,
      "locations" : [ {
        "id" : "StevensLoc",
        "street" : "Regency Drive",
        "zipCode" : "08817",
        "state" : "NJ"
      } ],
      "children" : 0,
      "salary" : 135000.0
    }, {
      "name" : "White",
      "age" : 32,
      "gender" : "Female",
      "maritalStatus" : "Married",
      "minor" : false,
      "locations" : [ {
        "id" : "WhiteLoc",
        "street" : "Green Road",
        "zipCode" : "33019",
        "state" : "FL"
      } ],
      "children" : 2,
      "salary" : 195000.0
    }, {
      "name" : "Smith",
      "age" : 46,
      "gender" : "Male",
      "maritalStatus" : "Single",
      "minor" : false,
      "locations" : [ {
        "id" : "SmithLoc",
        "street" : "Maple Street",
        "zipCode" : "90027",
        "state" : "CA"
      } ],
      "children" : 1,
      "salary" : 120000.0
    }, {
      "name" : "Green",
      "age" : 28,
      "gender" : "Female",
      "maritalStatus" : "Married",
      "minor" : false,
      "locations" : [ {
        "id" : "GreenLoc",
        "street" : "Green Road",
        "zipCode" : "33019",
        "state" : "FL"
      } ],
      "children" : 3,
      "salary" : 140000.0
    }, {
      "name" : "Brown",
      "age" : 32,
      "gender" : "Male",
      "maritalStatus" : "Married",
      "minor" : false,
      "locations" : [ {
        "id" : "BrownLoc",
        "street" : "Green Road",
        "zipCode" : "33019",
        "state" : "FL"
      } ],
      "children" : 2,
      "salary" : 65000.0
    }, {
      "name" : "Klaus",
      "age" : 54,
      "gender" : "Male",
      "maritalStatus" : "Married",
      "minor" : false,
      "locations" : [ {
        "id" : "KlausLoc",
        "street" : "Green Road",
        "zipCode" : "33019",
        "state" : "FL"
      } ],
      "children" : 1,
      "salary" : 85000.0
    }, {
      "name" : "Houston",
      "age" : 47,
      "gender" : "Female",
      "maritalStatus" : "Single",
      "minor" : false,
      "locations" : [ {
        "id" : "HoustonLoc",
        "street" : "Green Road",
        "zipCode" : "33019",
        "state" : "FL"
      } ],
      "children" : 2,
      "salary" : 135000.0
    }, {
      "name" : "Long",
      "age" : 29,
      "gender" : "Male",
      "maritalStatus" : "Married",
      "minor" : false,
      "locations" : [ {
        "id" : "LongLoc",
        "street" : "Green Road",
        "zipCode" : "33019",
        "state" : "FL"
      } ],
      "children" : 3,
      "salary" : 40000.0
    }, {
      "name" : "Short",
      "age" : 22,
      "gender" : "Male",
      "maritalStatus" : "Single",
      "minor" : false,
      "locations" : [ {
        "id" : "ShortLoc",
        "street" : "Green Road",
        "zipCode" : "33019",
        "state" : "FL"
      } ],
      "children" : 0,
      "salary" : 120000.0
    }, {
      "name" : "Doe",
      "age" : 21,
      "gender" : "Female",
      "maritalStatus" : "Single",
      "minor" : false,
      "locations" : [ {
        "id" : "DoeLoc1",
        "street" : "Green Road",
        "zipCode" : "33019",
        "state" : "FL"
      }, {
        "id" : "DoeLoc2",
        "street" : "Morgan Street",
        "zipCode" : "33020",
        "state" : "FL"
      }, {
        "id" : "DoeLoc3",
        "street" : "Lyric Ave",
        "zipCode" : "90027",
        "state" : "CA"
      } ],
      "children" : 1,
      "salary" : 210000.0
    } ],
    "selectedZipCode" : "08817"
  }
}'

please answer to the following questions:

1) What is the current total number of employees?
2a) How many children all employees have? 
2b) How many children does the average employee have?
3a) What is an average salary? 
3b) What is the maximal and minimal salaries?
4) How many employees are single?
5) In which states do the employee with "name" "Warner" have residences?
6a) How many people are inside 20% of highest paid employees? 
6b) Who are these high-paid employees?
7) List employees living in zipCode "33019".

Claude (https://claude.ai/)

1) The current total number of employees is 12.
2a) All employees have a total of 16 children
2b) The average employee has 1.33 children
3a) The average salary is $135416.67
3b) The maximum salary is $220,000.0 and the minimum salary is $40,000.0.
4) There are 4 single employees
5)The employee 'Warner' has residences in the following states: NJ
6a) There are 2 people in the top 20% of salaries
6b)The highest paid employees are:
 - Robinson ($220000.00)
 - Doe ($210000.00)
7) Employees living in zipcode 33019:
 - Robinson
 - White
 - Green
 - Brown
 - Klaus
 - Houston
 - Long
 - Short
 - Doe

ChatGPT (https://chat.openai.com/)

From the previous prompt I removed “without giving the source code”

1) Total number of employees: 12
2a) Total number of children all employees have: 16
2b) Average number of children per employee: 1.3333333333333333
3a) Average salary: 140416.66666666666
3b) Maximal salary: 220000.0
    Minimal salary: 40000.0
4) Number of single employees: 4
5) States where Warner has residences: ['NJ']
6a) Number of people inside 20% of highest-paid employees: 5
6b) High-paid employees: ['Robinson', 'Doe', 'White', 'Warner', 'Houston']
7) Employees living in zipCode 33019: ['Robinson', 'White', 'Green', 'Brown', 'Klaus', 'Houston', 'Long', 'Short', 'Doe']

Gemini (https://gemini.google.com)

From the previous prompt I removed “without giving the source code”

1) Total Employees:  There are 12 employees in the company.
2a) Total Children:  All employees have a total of 18 children (2+0+0+2+1+3+2+1+2+3+0+1).
2b) Average Children:  There's an average of 1.5 children per employee (18 children / 12 employees).
3a) Average Salary:  The average salary is $141,666.67 (sum of all salaries / number of employees).
3b) Min & Max Salary:  The maximum salary is $220,000 (Robinson) and the minimum salary is $40,000 (Long).
4) There are 4 single employees (Stevens, Smith, Houston, Short).
5) Warner has a residence only in New Jersey (NJ) based on the "state" field in his location details.
6a) Number:  The top 2 employees (20% of 12) are in the highest salary range.
6b) Employees: The highest-paid employees are Robinson ($220,000) and Doe ($210,000).
7) Employees in Zip Code "33019":  These employees live in zip code "33019":
    White
    Green
    Brown
    Klaus
    Houston
    Long
    Doe (one of their residences)

My python code

emps = json_data['company']['employees']
# 1) 
len(emps)  12
# 2a
sum(map(lambda x: x['children'], emps)) # 17
# 2b)
sum(map(lambda x: x['children'], emps)) / 12 # 1.416
# 3a)
sum(map(lambda x: x['salary'], emps)) / 12.0 # 134583.3
# 3b)
max(map(lambda x: x['salary'], emps)) # 220000.0
min(map(lambda x: x['salary'], emps)) # 40000.0
# 4)
len(list(filter(lambda x: x['maritalStatus'] == 'Single', emps)))
# 5)
list(filter(lambda x: x['name'] == 'Warner', emps))[0]['locations'] # [{'id': 'WarnerLoc', 'street': 'Maple Street', 'zipCode': '08817', 'state': 'NJ'}]
# 6b)
list(map(lambda x: x['name'], sorted(emps, key=lambda x: x['salary'], reverse=True)[0:3])) # ['Robinson', 'Doe', 'White']

Calling http rest API and parsing json response with (scryer) #Prolog

Sat, 02 Mar 2024 00:00:00 +0000

In this simple code I’ll show how you can call a rest api (qliksense saas rest api) and parse its json response


?- use_module(library(dcgs)).
?- use_module(library(pairs)).
?- use_module(library(serialization/json)).
?- use_module(library(http/http_open)).

?- set_prolog_flag(answer_write_options, [max_depth(1000)]).

?- http_open("https://mytenant.eu.qlikcloud.com/api/v1/tenants/me", S, [request_headers(['Authorization'("Bearer MYKEY")])]), 
	get_n_chars(S, _N, HTML),
	phrase(json_chars(pairs(J)), HTML), 
	member(string("id")-string(ID), J).

 HTML = "{\"id\":\"myid\",\"hostnames\":[\"mytenant.eu.qlikcloud.com\"]}", J = [string("id")-string("myid"),string("hostnames")-list([string("mytenant.eu.qlikcloud.com)])], ID = "myid"

Ragno: Top Websites Ranking

Sat, 10 Feb 2024 00:00:00 +0000

Ragno is a new web crawler! It is still in alpha version. Below its top website ranking

60490 facebook.com
52334 twitter.com
45800 instagram.com
37198 youtube.com
28622 linkedin.com
10613 t.me
9388 github.com
6891 wordpress.org
5211 vk.com
5134 pinterest.com
4702 google.com
3959 tiktok.com
3573 play.google.com
2972 bit.ly
2868 wordpress.com
2572 apps.apple.com
2476 goo.gl
2467 youtu.be
2439 wa.me
2426 en.wikipedia.org
2388 subscribe.wordpress.com
2384 en.wordpress.com
2177 discord.gg
2133 vimeo.com
2050 amazon.com
2033 open.spotify.com
1986 reddit.com
1949 creativecommons.org
1916 api.whatsapp.com
1862 beian.miit.gov.cn

Apache Httpd server

Fri, 12 Jan 2024 00:00:00 +0000

Apache Httpd The Apache HTTP Server is a free and open-source cross-platform web server software, released under the terms of Apache License 2.0. It is developed and maintained by a community of developers under the auspices of the Apache Software Foundation.

Some web sites using the opensource software Apache httpd are:

Contacts

Fri, 12 Jan 2024 00:00:00 +0000

My name is Matteo Redaelli, I live in Italy next to Lecco

You can contact me via email at matteo.redaelli@gmail.com or twitter

Regards

Family Riddle problem with Prolog

Fri, 12 Jan 2024 00:00:00 +0000

My #prolog solution for “Family Riddle” proposed by dmcommunity.org challenge Jan 2024.

Let’s assume that Rene and Leo are both heads of household, and, what a coincidence, both families include three girls and three boys. The youngest child in Leo’s family is a girl, and in Rene’s family, a little girl has just arrived. In other words, there is a girl in Rene’s family whose age is less than one year. Neither family includes any twins, nor any children closer in age than a year. All the children are under age ten. In each family, the sum of the ages of the girls is equal to the sum of the ages of the boys; in fact, the sum of the squares of the ages of the girls is equal to the sum of the squares of the ages of the boys. The sum of the ages of all these children is 60.

My solution (without duplicates / permutations): the ages of children of the same sex are ordered.

?- findall([LeoBoys,LeoGirls,ReneBoys,ReneGirls], sol([LeoBoys,LeoGirls,ReneBoys,ReneGirls]), Solutions), length(Solutions, Tot).
Solutions = [[[4, 5, 9], [3, 7, 8], [1, 3, 8], [0, 5, 7]]],
Tot = 1.

Below the code


max_age(9).
sum_age(60).
count_same_sex(3).

constraints_family_same_sex(L):-
    max_age(MaxAge),
    count_same_sex(N),
    %% both families include three girls and three boys.
    length(L, N),
    %% All the children are under age ten
    L ins 0..MaxAge,
    %% ages in sorted order 
    ordered_list(L).

constraints_family(Boys, Girls):-
    sum_list(Boys, Sum),
    sum_list(Girls, Sum),
    sum_square_list(Boys, SumS),
    sum_square_list(Girls, SumS).

sum_list([E|L], Sum):-
    sum_list(L, SumL),
    Sum #= E + SumL.
sum_list([], 0).

sum_square_list([E|L], Sum):-
    sum_square_list(L, SumL),
    Sum #= E ^ 2 + SumL.
sum_square_list([], 0).

ordered_list([E1,E2|L]):-
    E1 #=< E2,
    ordered_list([E2|L]).
ordered_list([_E]).
ordered_list([]).

sol([LeoBoys, LeoGirls, ReneBoys, ReneGirls]):-    
    %% All the children are under age ten
    %% both families include three girls and three boys.
    constraints_family_same_sex(LeoBoys),
    constraints_family_same_sex(LeoGirls),
    constraints_family_same_sex(ReneBoys),
    constraints_family_same_sex(ReneGirls),

    %% in Rene’s family, a little girl has just arrived
    ReneGirls = [0|_],

    %% The youngest child in Leo’s family is a girl
    LeoBoys = [LB1|_RestLeoBoys],
    LeoGirls = [LG1|_RestLeoGirls],
    LG1 #< LB1,

    append(LeoBoys, LeoGirls, LeoList),
    append(ReneBoys, ReneGirls, ReneList),
    append(LeoList, ReneList, AllChildren),
    
    %% Neither family includes any twins, nor any children closer in age than a year
    all_distinct(LeoList),
    all_distinct(ReneList),
    
    %% The sum of the ages of all these children is 60.
    sum_age(SumAge),
    sum_list(AllChildren, SumAge),


    %% In each family, the sum of the ages of the girls is equal to the sum of the ages of the boys
    constraints_family(LeoBoys, LeoGirls),
    constraints_family(ReneBoys, ReneGirls),
    
    label(AllChildren).

Nginx web server

Fri, 12 Jan 2024 00:00:00 +0000

Nginx is a web server that can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache. The software was created by Russian developer Igor Sysoev and publicly released in 2004.[9] Nginx is free and open-source software, released under the terms of the 2-clause BSD license. A large fraction of web servers use Nginx,[10] often as a load balancer

Some web sites using the opensource software Nginx are:

Opensource softwares - database

Fri, 12 Jan 2024 00:00:00 +0000

Search engines

TODO

RDBMS

Opensource softwares - Prolog

Fri, 12 Jan 2024 00:00:00 +0000

SWI-Prolog: homepage, repo
GNU Prolog: homepage, repo
YAP Prolog: homepage, repo
B-Prolog: homepage, repo
Ciao Prolog: homepage, repo
Scryer Prolog: homepage, repo
XSB Prolog: homepage, repo
ECLiPSe Prolog: homepage, repo
Tau Prolog: homepage, repo
Logtalk: homepage, repo
tuProlog: homepage, repo
Strawberry Prolog: homepage, repo
Lean Prolog: homepage, repo
Trealla: homepage, repo
N-Prolog: homepage, repo

Varnish cache

Fri, 12 Jan 2024 00:00:00 +0000

Varnish cache is a reverse caching proxy used as HTTP accelerator for content-heavy dynamic web sites as well as APIs. In contrast to other web accelerators, such as Squid, which began life as a client-side cache, or Apache and nginx, which are primarily origin servers, Varnish was designed as an HTTP accelerator. Varnish is focused exclusively on HTTP, unlike other proxy servers that often support FTP, SMTP, and other network protocols.

Some web sites using the opensource software Varnish are:

Wordpress

Fri, 12 Jan 2024 00:00:00 +0000

Wordpress WordPress (also known as WP or WordPress.org) is a web content management system. It was originally created as a tool to publish blogs but has evolved to support publishing other web content, including more traditional websites, mailing lists and Internet forum, media galleries, membership sites, learning management systems and online stores. Available as free and open-source software, WordPress is among the most popular content management systems – it was used by 43.1% of the top 10 million websites as of December 2023

Some web sites using the opensource software Wordpress are:

Coins challenge Oct 2023 with Prolog

Sun, 12 Nov 2023 00:00:00 +0000

My #prolog solution for “Coins” proposed by dmcommunity.org challenge Oct 2023.

Suppose you need to pay 1 Euro. In how many different ways you can do it via 1c, 2c, 5c, 10c, 20c, 50c and 1 Euro coins?

%%  findall(S, sol(100, S), All), maplist(writeln, All).

%% ...
%% [94,3,0,0,0,0,0]
%% [95,0,1,0,0,0,0]
%% [96,2,0,0,0,0,0]
%% [98,1,0,0,0,0,0]
%% [100,0,0,0,0,0,0]
%% ...

Below the code


:- use_module(library(clpfd)).

sol(TOT, [C1, C2, C5, C10, C20, C50, C100]) :-
    C1   in 0..100,
    C2   in 0..50,
    C5   in 0..20,
    C10  in 0..10,
    C20  in 0..5,
    C50  in 0..2,
    C100 in 0..1,
    TOT #= C1*1 + C2*2 + C5*5 + C10*10 + C20*20 + C50*50 + C100*100,
    label([C1, C2, C5, C10, C20, C50, C100]).

DevOps Qlik Cloud - managing qlik cloud resources with OpenTofu / Terraform

Tue, 10 Oct 2023 09:14:35 +0000

Abstract

Open tofu (a Terraform fork) is an open-source infrastructure as code (IaC) tool that allows you to define, provision, and manage infrastructure as code. Terraform can be used to manage a wide range of infrastructure resources, including Qlik Cloud resources.

Here are some of the benefits of using OpenTofu/Terraform to manage Qlik Cloud resources:

Consistency: Terraform allows you to define your infrastructure in a consistent and repeatable way. This can help to reduce errors and improve the overall quality of your infrastructure.

Automation: Terraform can be used to automate the deployment and management of your Qlik Cloud resources. This can save you time and effort, and it can also help to improve the reliability of your infrastructure.

Version control: Terraform allows you to version control your infrastructure configuration. This means that you can track changes to your infrastructure over time and easily roll back to previous versions if necessary.

To manage Qlik Cloud resources with Terraform, you will need to use the Mastercard/terraform-provider-restapi provider. This provider allows you to use Terraform to manage any REST API, which makes it very flexible and powerful.

Usage

Below I’ll show how to create some Qlik groups, spaces and space assigments.

Create a folder and inside the following files:

File variables.tf

variable "QLIK_APIKEY" {
  type = string
}

variable "QLIK_URI" {
  type = string
}

File versions.tf

terraform {
  required_providers {
    restapi = {
      source = "mastercard/restapi"
      version = "1.18.2"
    }
  }
}

File providers.tf

provider "restapi" {
  alias                = "qlik_saas"
  uri                  = "${var.QLIK_URI}"
  debug                = true
  write_returns_object = true

  headers = {
      Authorization = "Bearer ${var.QLIK_APIKEY}"
      "Content-Type" = "application/json"
  }
}

File qlik_groups.tf

resource "restapi_object" "group_Developer" {
    provider = restapi.qlik_saas
    path = "/api/v1/groups"
    data = jsonencode({
        name: "Developer"
        status: "active"
        assignedRoles: []})
}

resource "restapi_object" "group_Italy" {
    provider = restapi.qlik_saas
    path = "/api/v1/groups"
    data = jsonencode({
        name: "Italy"
        status: "active"
        assignedRoles: []})
}

File qlik_space_italy.tf

resource "restapi_object" "space_Italy" {
    provider = restapi.qlik_saas
    path = "/api/v1/spaces"
    data = jsonencode({
	name: "Italy"
	type: "managed"
	description: "Italy space" })
}

resource "restapi_object" "space_Italy_assigments_group_Developer" {
    provider = restapi.qlik_saas
    path = "/api/v1/spaces/${resource.restapi_object.space_Italy.id}/assignments"
    data =  jsonencode({
	type: "group"
	roles: ["consumer","contributor","dataconsumer","publisher"]
	assigneeId: "${resource.restapi_object.group_Developer.id}"
    })
}

resource "restapi_object" "space_Italy_assigments_group_Italy" {
    provider = restapi.qlik_saas
    path = "/api/v1/spaces/${resource.restapi_object.space_Italy.id}/assignments"
    data =  jsonencode({
	type: "group"
	roles: ["consumer","dataconsumer"]
	assigneeId: "${resource.restapi_object.group_Italy.id}"
    })
}

Now you can create your resources with

export TF_VAR_QLIK_APIKEY=xxxxxxxxxxxxxxxxxxxxxxxx
export TF_VAR_QLIK_URI=https://mytenant.eu.qlikcloud.com

tofu init
tofu plan -out myplan
tofu apply myplan

Importing existing resources

You could also import in the terraform state any resources already/manually created with something like

tofu import restapi_object.space_Germany /api/v1/spaces/12222222

Managing several environments/states

tofu workspace create dev
tofu workspace create prd
tofu workspace select dev

References:

qlik-cloud-devops-terraform-opentofu github repository
OpenTofu witch is a fork of Terraform under the Linux Foundation unbrella.
Mastercard/terraform-provider-restapi
Qlik cloud

Calling Qliksense Repository API from Apache Drill via sql

Wed, 23 Feb 2022 00:00:00 +0000

Abstract

I’ll show how to connect to Qliksense Repository API via sql using Apache Drill.

In this example Qliksense engine service runs at https://qlik.redaelli.org:4242/

Download

Download and unzip Apache Drill from https://drill.apache.org/download/

Configure

Create or edit the file conf/storage-plugins-override.conf

"storage": {
  "qliksense" : {
    "type" : "http",
    "cacheResults" : true,
    "connections" : {
      "p4242" : {
	"url" : "http://nginx.redaelli.org/qlik/",
	"method" : "GET",
	"authType" : "none",
	"userName" : null,
	"password" : null,
	"postBody" : null,
	"params" : ["filter"],
	"dataPath" : null,
	"requireTail" : true,
	"inputType" : "json",
	"xmlDataLevel" : 1
      }
    },
    "proxyType" : "direct",
    "enabled" : true
  }
}

At the moment Apache drill cannot query external rest services with client certificates DRILL-8052 and so we need a reverse proxy like Nginx.

Inside nginx configure a reverse proxy like

    location /qlik/ {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header x-qlik-xrfkey 9003456789Zbcdez;
        proxy_set_header X-Qlik-User "UserDirectory=internal;UserId=sa_repository";
        set $args $args&xrfkey=9003456789Zbcdez;
        proxy_pass https://qlik.redaelli.org:4242/;
        proxy_ssl_certificate     /certificates/qlik/client.pem;
        proxy_ssl_certificate_key /certificates/qlik/client_key.pem;
        proxy_ssl_verify       off;
    }

Start Drill

Run drill with

./bin/drill-embedded

Now the system is started and you have a command line sql interface. There is also a web ui at http://localhost:8047/ where you can run queries or add/change storages/connections instead of editing the previous file (http://localhost:8047/storage)

Queries

Extract a stream with custom properties values (nested json)

select
  t1.name as streamName,
  t1.owner as streamOwner,
  t1.cps.definition.name as cpName,
  t1.cps.value as cpValue
from (
  select
    s.name,
    s.owner.name owner,
    flatten(s.customProperties) as cps
  from
    qliksense.p4242.`qrs/stream/full` s
  where
    filter='name eq ''Merlot'''
) t1;

The output is

+------------+---------------+---------------+-----------------+
| streamName |  streamOwner  |    cpName     |     cpValue     |
+------------+---------------+---------------+-----------------+
| Merlot     | sa.vino       | CustomBalance | SelfService     |
| Merlot     | sa.vino       | GroupAccess   | Qliksense_Merlot|
+------------+---------------+---------------+-----------------+

Calling Talend Cloud Rest API from Apache Drill via sql

Fri, 04 Feb 2022 00:00:00 +0000

Abstract

I’ll show how to connect to Talend Cloud API via sql using Apache Drill.

Download

Download Apache Drill from https://drill.apache.org/download/

Configure

Create or edit the file conf/storage-plugins-override.conf

"storage": {
  "talendcloud" : {
	"type" : "http",
	"cacheResults" : true,
	"connections" : {
	  "get" : {
		"url" : "https://api.eu.cloud.talend.com",
		"method" : "GET",
		"headers" : {
		  "Authorization" : "Bearer MYSECRETKEY"
		},
		"authType" : "none",
		"userName" : null,
		"password" : null,
		"postBody" : null,
		"params" : ["query"],
		"dataPath" : null,
		"requireTail" : true,
		"inputType" : "json",
		"xmlDataLevel" : 1
	  }
	},
	"proxyType" : "direct",
	"enabled" : true
  }
}

Start Drill

Run drill with

./bin/drill-embedded

Queries

Clusters and remote engines

alter session set `store.json.all_text_mode` = true;

select
	engine.*, clusters.*
from
	talendcloud.get.`/tmc/v2.6/runtimes/remote-engines` as engine
inner join
	talendcloud.get.`/tmc/v2.6/runtimes/remote-engine-clusters` as clusters
on engine.clusterId = clusters.id
where
	engine.query='workspace.environment.name==PRD;status==PAIRED;workspace.name==DATA' limit 1;

The output is

createDate               2021-06-10T09:40:37.676Z
updateDate               2021-06-10T10:09:47.780Z
id                       646464
name                     AWS_PRD_02
workspace                {"id":"8886","name":"DATA","owner":"matteo","type":"custom","environment":{"id":"8787","name":"PRD","default":"false"}}
runtimeId                4434
availability             AVAILABLE
status                   PAIRED
runProfiles              ["TALEND_RUNTIME","MICROSERVICE","JOB_SERVER"]
debug                    {}
clusterId                6565
esbCompatibilityVersion  7.3.1
description              null
createDate0              2021-06-30T15:16:43.538Z
updateDate0              2021-06-30T15:16:43.538Z
id0                      5454
name0                    AWS_CLUSTER
description0             Remote Engine Cluster - Produzione AWS
workspace0               {"id":"32323","name":"DATA","owner":"matteo","type":"custom","environment":{"id":"5454","name":"PRD","default":"false"}}
runtimeId0               1112
availability0            AVAILABLE

Users and groups

From the TMC it id not possibile to extract a csv with users and groups. The following queries can help.

select
	g.name,
	u.email
from
	(select name, flatten(userIds) as uid
	from talendcloud.get.`v1/management/groups`) g
inner join
	  talendcloud.get.`v1/management/users` u
on
	g.uid = u.id
order by
	g.name,
	u.mail;

with users as
	 (select email, flatten(roles) role from talendcloud.get.`v1/management/users`)
select
	users.email as email,
	users.role.name as role
from users
where
	users.role.name like 'COMPANY1%' or users.role.name like 'COMPANY2%';

Sharing Windows guest vpn connection

Sun, 03 Oct 2021 00:00:00 +0000

Abstract

I have a linux workstation with a corporate guest Windows image running on kvm. I would like to share the guest windows vpn connection with my linux host workstation.

My linux host has one network: virbr0 (public) with an ip 192.168.122.x

My vpn/corporate subnet is 10.150.0.0/16

Solution

Create on linux/kvm a new host only/private net ( I have an ip 192.168.100.x) and enable it in the windows guest os (ip 192.168.100.236)

On the guest OS you should have several networks. I named them: virbr0 (public/internet), virbr1 (private, host only), vpn (created/used by your vpn client).

Solution with sshuttle

(sshuttle): connecting to many remote hosts like native connections (without socks and pac files).

“Sshuttle is not exactly a VPN, and not exactly port forwarding. It’s kind of both, and kind of neither.”

“Sshuttle assembles the TCP stream locally, multiplexes it statefully over an ssh session, and disassembles it back into packets at the other end. So it never ends up doing TCP-over-TCP. It’s just data-over-TCP, which is safe.”

I installed sshuttle with “apt-get install sshuttle” on my debian laptop

Then I installed sshd and python on the guest windows (I used cygwin).

Finally I was able to run:

cp /etc/resolv.conf.corporate /etc/resolv.conf
sshuttle -r matteo@192.168.100.239 10.150.0.0/16

Guest Windows side

Start your vpn in your windows guest OS (I saw that checkpoint - I suppose for securty reason - changes the ip address of the host net and share options of its own net)

Share the vpn net with the host only net: below my screenshots

Host Linux side

Enable the required routing

cp /etc/resolv.conf.corporate /etc/resolv.conf
ip route add 10.150.0.0/15 via 192.168.100.236 dev virbr1

PS

One year ago I wrote an article with some useful networking tricks https://matteoredaelli.netlify.app/news/my-networking-survival-kit/

Parsing qliksense healthcheck api results

Wed, 15 Sep 2021 00:00:00 +0000

Abstract

When you do a stress test/troubleshooting of a qliksense node it is useful to collect the responses of the healthcheck api and extract some useful info from them (which and how many applications were loaded in memory, …)

Collecting data

I usually use the command line tool qsense for querying the Qliksense repository

while [ 1 ]
do 
    qsense healthcheck  qlikhost1.redaelli.org ~/certificates/qlik/client.pem >> healthcheck.jl
    sleep 60
done

Each line of the file healthcheck.jl is a json object like

{
  "version": "12.763.10",
  "started": "20210915T165938.000+0200",
  "mem": {
    "committed": 72283.234375,
    "allocated": 118266.04296875,
    "free": 436586.79296875
  },
  "cpu": {
    "total": 0
  },
  "session": {
    "active": 1,
    "total": 63
  },
  "apps": {
    "active_docs": [
      "0599e6baa-3b4a-4648-bbdb-47013a02dc21"
    ],
    "loaded_docs": [
      "15e0547c-c4eb-4492-a1db-1603d8295423",
      "163777f8-9582-46b0-9418-a01f2d71c32d",
      "059e6baa-3b4a-4648-bbdb-47013a02dc21"
    ],
    "in_memory_docs": [
      "15e0547c-c4eb-4492-a1db-1603d8295423",
      "163777f8-9582-46b0-9418-a01f2d71c32d",
      "059e6baa-3b4a-4648-bbdb-47013a02dc21"
    ],
    "calls": 17126,
    "selections": 300
  },
  "users": {
    "active": 1,
    "total": 6
  },
  "cache": {
    "hits": 70,
    "lookups": 70,
    "added": 0,
    "replaced": 0,
    "bytes_added": 0
  },
  "saturated": false,

It can be useful to show application names instead of their Ids. So we can download the dataset with

qsense entity qlikhost1.redaelli.org ~/certificates/qlik/client.pem app --filter "published eq true" > app.json

Extracting some info

With the following script

python healthcheck.py healthcheck.jl healthcheck-out

you can extract a csv with the info

timestamp, mem_free, session_active, session_total, users_active, users_total, app1, app2, app3, ...

Below the source of the script

## file  healthcheck.py
import sys

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

infile = sys.argv[1]
outfile = sys.argv[2]

spark = SparkSession.builder.getOrCreate()

df = spark.read.json(infile)

apps = spark.read.json("apps.json")

df.select(F.col("now"), 
          F.col("mem.free").alias("mem_free"), 
          F.col("session.active").alias("session_active"), 
          F.col("session.total").alias("session_total"),
          F.col("users.active").alias("users_active"), 
          F.col("users.total").alias("users_total"),
          F.col("apps.in_memory_docs"))\
    .withColumn("id", F.explode(F.col("in_memory_docs")))\
    .join(apps, 'id', how="left").withColumn("fullname", F.concat("name", "id")).select(["now", "mem_free", "session_active", "session_total","users_active", "users_total", "fullname"])\
    .groupBy(["now", "mem_free", "session_active", "session_total","users_active", "users_total"])\
      .pivot("fullname").count().coalesce(1).write.mode("overwrite").option("sep",";").option("header","true").csv(outfile)

When/how many times was the engine restarted? What happened just before?

import collections
import pprint
import re
import pandas as pd
import sys

def load_data(day, infile):
    return pd.read_json(infile, lines=True)

def load_apps():
    url = "app.json"
    return pd.read_json(url)
    
def parse_healtcheck(infile):
    def lists_diff(li1, li2):
        return list(set(li1) - set(li2)) + list(set(li2) - set(li1))
    
    df = load_data(infile)
    started = None
    active = loaded = last_active = last_loaded = []
    for index, row in df.iterrows():
        if row["started"] != started:
            print("******************************************")
            print("Engine started at {started}".format(started=row["started"]))
            print("******************************************")
            started=row["started"]
            print("Previous active:")
            print(active)
            print("Previous loaded:")
            print(loaded)
            last_active = last_active + active
            last_loaded = last_loaded + loaded
        new_active = row["apps"]["active_docs"]
        new_loaded = row["apps"]["loaded_docs"]
    
        delta_active = lists_diff(new_active, active)
        if delta_active != []:
            pprint.pprint(row["now"] + ": new active apps: " + str(delta_active))

        delta_loaded = lists_diff(new_loaded, loaded)
        if delta_loaded != []:
            pprint.pprint(row["now"] + ": new loaded apps: " + str(delta_loaded))
        
        active = new_active
        loaded = new_loaded
    pprint.pprint("Latest loaded apps: " + str(collections.Counter(last_loaded)))
    pprint.pprint("Latest active apps: " + str(collections.Counter(last_active)))
    pprint.pprint("Latest restarts: " + str(df.started.unique()))
    
if __name__ == "__main__":
    # execute only if run as a script
    infile = sys.argv[1]
    parse_healthcheck(infile)

aws-ext

Fri, 25 Jun 2021 00:00:00 +0000

The aws_ext python package contains some useful functions (built on top of boto3) for managing some aws services. At the moment only some utilities for the Aws Glue Data catalog

Installation

pip install aws_ext

Usage

import boto3
import aws_ext

session = boto3.session.Session()

GLUE

from aws_ext import glue_databases
glue_client = session.client("glue")

Extracting tables with (too) many versions

glue_databases.get_tables_with_many_versions(glue_client, database_name="mydb", threshold=1)

Deleting old tables versions

glue_databases.delete_old_tables_versions(glue_client, database_name="mydb", keep=1, dryrun=True)

Where is gold challenge with Prolog

Wed, 02 Jun 2021 00:00:00 +0000

My #prolog solution for “Where is gold?” proposed by dmcommunity.org challenge Jun 2021: 0 means empty, 1 means gold

?- solution(Box1,Box2,Box3).
Box1 = Box3, Box3 = 0,
Box2 = 1.

below the code

:-use_module(library(clpfd)).

sentence1(    1,_Box2,_Box3).
not_sentence1(0,_Box2,_Box3).

sentence2(    _Box1,0,_Box3).
not_sentence2(_Box1,1,_Box3).

sentence3(    0,_Box2,_Box3).
not_sentence3(1,_Box2,_Box3).

true_only_one_sentence(Box1, Box2, Box3):-
	(       sentence1(Box1,Box2,Box3), not_sentence2(Box1,Box2,Box3), not_sentence3(Box1,Box2,Box3) ) ;
	(   not_sentence1(Box1,Box2,Box3),     sentence2(Box1,Box2,Box3), not_sentence3(Box1,Box2,Box3) ) ;
	(   not_sentence1(Box1,Box2,Box3), not_sentence2(Box1,Box2,Box3),     sentence3(Box1,Box2,Box3) ).

solution(Box1, Box2, Box3):-
	Box1 in 0..1, /* 0 empty, 1 gold */
	Box2 in 0..1,
	Box3 in 0..1,
	Box1 + Box2 + Box3 #= 1, /* only one box contains gold */
	true_only_one_sentence(Box1, Box2, Box3).

GraphQL datasource for Qliksense

Sat, 27 Mar 2021 00:00:00 +0000

Qliksense datasource for GraphQL Apollo server

apollo-datasource-qliksense is a Qliksense datasource for the GraphQL Apollo server. “Apollo is the industry-standard GraphQL implementation, providing the data graph layer that connects modern apps to the cloud.”

Installation

npm install apollo-datasource-qliksense

Usage

Below a very basic sample

File index.js file:

const resolvers = require('./resolvers');

const { ApolloServer, gql } = require('apollo-server');
const { makeExecutableSchema } = require('graphql-tools');
const { QliksenseDataSource } = require('apollo-datasource-qliksense');

const qliksense = new QliksenseDataSource("https://myqlikserver.redaelli.org:4242",
				process.env.QLIK_CERTIFICATE,
				process.env.QLIK_CERTIFICATE_KEY);

const { readFileSync } = require('fs')

const typeDefs = readFileSync('./schema.graphql).toString('utf-8')

const graphqlSchemaObj = { //makeExecutableSchema({
	typeDefs: typeDefs,
	resolvers: resolvers,
	tracing: true,
	dataSources: () => ({ qliksense: qliksense })
};

const server = new ApolloServer(graphqlSchemaObj);

// The `listen` method launches a web server.
server.listen().then(({ url }) => {
  console.log(`🚀  Server ready at ${url}`);
});

File schema.graphql

type Json {
	 json: String
	 }

type Query {
  qliksense_entity(
	entity: String!,
	count: Boolean,
	filter:String): Json
}

File resolvers.js

module.exports = {
  Query: {
	  qliksense_entity: async (_source, { entity, count, filter }, { dataSources }) => {
	  resp = await dataSources.qliksense.getEntity(entity, count, filter);
	  return {json:  JSON.stringify(resp) }
	  },
  },
}

Sample graphql queries

{
  qliksense_entity(
	entity: "custompropertydefinition"
	filter: "Name eq 'GroupAccess'"
  ) {
	json
  }
}

{
  qliksense_entity(
	entity: "user"
	count: true
  ) {
	json
  }
}

Graphql schema for Qliksense entities

I suggest the tool https://github.com/walmartlabs/json-to-simple-graphql-schema for building a graphql schema from a sample json file (an export of a Qliksense entity like app, user, custompropertydefinition, datasource, …).

qsense python library and command line tool for qliksense

Tue, 16 Mar 2021 00:00:00 +0000

qSense

qsense is an python library and command line tool for qliksense

Some useful commands

export_remove_old_apps

Export (published or passing any other filter) applications to qvd files

qsense export_delete_old_apps qliksense.redaelli.org ~/certificates/client.pem  --target_path '/tmp' --modified_days=300 --last_reload_days=300

deallocate_analyzer_licenses_for_professionals

Deallocate analyzer license fom users with a professional license

qsense deallocate_analyzer_licenses_for_professionals qliksense.redaelli.org ~/certificates/client.pem --nodryrun

delete_removed_exernally_users

Delete users that were removed externally (from active directory?)

qsense delete_removed_exernally_users qliksense.redaelli.org ~/certificates/client.pem GROUP --nodryrun

Medical Services challenge with Prolog

Tue, 02 Feb 2021 00:00:00 +0000

My #swi #prolog solution for “Benchmark ‘Medical Services’” proposed by dmcommunity.org challenge Feb 2021 can be tested running a public docker image

	docker run -p 8888:8888 -d --name matteoredaelli/dmcommunity_org_2021_02:latest
	wget https://github.com/DMCommunity/dmcommunity_shared/raw/master/MedicalServices.json
	curl -XPOST -d @MedicalServices.json -H "Accept: application/json" -H "Content-type: application/json" http://localhost:8888/many

The decision table is a set of prolog facts like

decision_table('Office','acupuncture','PL123','L','Y','N','2015-01-01','2023-12-31','N','N','N').
decision_table('Outpatient','acupuncture','PL123','L','Y','N','2015-01-01','2023-12-31','N','N','N').
decision_table('Inpatient','acupuncture','PL123','L','N','N','2015-01-01','2023-12-31','N','N','N').

The core of the solution is inside the rules.pl file

parse_input(json([placeOfService=PlaceOfService,
		  type=Type,
		  plan=Plan,
		  groupSize=GroupSize,
		  inNetwork=InNetwork,
		  isCovered=IsCovered,
		  dateOfService=DateOfService,
		  coveredInFull=_,
		  copay=_,
		  coInsurance=_]),
		json([placeOfService=PlaceOfService,
		  type=Type,
		  plan=Plan,
		  groupSize=GroupSize,
		  inNetwork=InNetwork,
		  isCovered=IsCovered,
		  dateOfService=DateOfService,
		  coveredInFull=CoveredInFull,
		  copay=Copay,
		  coInsurance=CoInsurance])):-
	decision_table(PlaceOfService,
			   Type,
			   Plan,
			   GroupSize,
			   InNetwork,
			   IsCovered,
			   DateOfService1,
			   DateOfService2,
			   CoveredInFull,
			   Copay,
			   CoInsurance),
	%% check date: must be between the two dates in decision_table
	atom_string(DateOfService, DateOfServiceString),
	atom_string(DateOfService1, DateOfService1String),
	atom_string(DateOfService2, DateOfService2String),
	DateOfService1String @=< DateOfServiceString,
	DateOfServiceString  @=< DateOfService2String.

The file server.pl contains the code for the rest server that exposes the core functions as http POST requests.

References:

Virtual Chess tournament with Prolog

Mon, 07 Dec 2020 00:00:00 +0000

Below my #swi #prolog solution for “Virtual Chess tournament” proposed by dmcommunity.org challenge Dec 2020

Player 1 is Fischer, player 2 is Kasparov and player 3 is Karpov.

I can find two solutions (Score1,Wins1,Draws1,Loses1 are Fischer’s score, number of wins, draws and loses,..) is:

?- time(solution([[Score1,Wins1,Draws1,Loses1],[Score2,Wins2,Draws2,Loses2],[Score3,Wins3,Draws3,Loses3]])).

% 1,297,058 inferences, 0.136 CPU in 0.136 seconds (100% CPU, 9557549 Lips)
Score1 = 15,
Wins1 = 4,
Draws1 = 7,
Loses1 = Draws2, Draws2 = 3,
Score2 = 13,
Wins2 = 5,
Loses2 = 6,
Score3 = 14,
Wins3 = Loses3, Loses3 = 2,
Draws3 = 10 ;

% 152,142 inferences, 0.021 CPU in 0.021 seconds (100% CPU, 7203556 Lips)
Score1 = 15,
Wins1 = Draws2, Draws2 = 4,
Draws1 = 7,
Loses1 = 3,
Score2 = 14,
Wins2 = Loses2, Loses2 = 5,
Score3 = 13,
Wins3 = 1,
Draws3 = 11,
Loses3 = 2 ;

?- time(findall([[Score1,Wins1,Draws1,Loses1],[Score2,Wins2,Draws2,Loses2],[Score3,Wins3,Draws3,Loses3]], solution([[Score1,Wins1,Draws1,Loses1],[Score2,Wins2,Draws2,Loses2],[Score3,Wins3,Draws3,Loses3]]), L)), length(L, TotSolutions).

% 2,401,046 inferences, 0.252 CPU in 0.252 seconds (100% CPU, 9530423 Lips)
L = [[[15, 4, 7, 3], [13, 5, 3, 6], [14, 2, 10, 2]], [[15, 4, 7, 3], [14, 5, 4, 5], [13, 1, 11, 2]]],
TotSolutions = 2.

Below my dec2020.pl script

tot_games(14).
tot_games_two_players(7).

match_two_players(Wins1, Draws, Wins2):-
	%% constraint: 7 games agains each player
	tot_games_two_players(TotGames),
	Wins1 in 0..TotGames,
	Draws in 0..TotGames,
	Wins2 in 0..TotGames,
	Wins1 + Draws + Wins2 #= TotGames.

tournament([[Score1, Wins1, Draws1, Loses1],
	   [Score2, Wins2, Draws2, Loses2],
	   [Score3, Wins3, Draws3, Loses3]
	  ]):-
	tot_games(TotGames),
	match_two_players(Wins12, Draws12, Wins21),
	match_two_players(Wins13, Draws13, Wins31),
	match_two_players(Wins23, Draws23, Wins32),
	Wins1 #= Wins12 + Wins13,
	Wins2 #= Wins21 + Wins23,
	Wins3 #= Wins31 + Wins32,
	Loses1 #= Wins21 + Wins31,
	Loses2 #= Wins12 + Wins32,
	Loses3 #= Wins13 + Wins23,
	Draws1 #= Draws12 + Draws13,
	Draws2 #= Draws12 + Draws23,
	Draws3 #= Draws13 + Draws23,

	Wins1 + Draws1 + Loses1 #= TotGames,
	Wins2 + Draws2 + Loses2 #= TotGames,
	Wins3 + Draws3 + Loses3 #= TotGames,

	Score1 #= Wins1 * 2 + Draws1,
	Score2 #= Wins2 * 2 + Draws2,
	Score3 #= Wins3 * 2 + Draws3,

	%% constraint: Fischer (1) is the winner
	Score1 #> Score2,
	Score1 #> Score3,

	%% constraint: Kasparov (2) is the most aggressive
	Wins2 #> Wins1,
	Wins2 #> Wins3,

	%% constraint: Karpov (3) is the best defensive player
	Loses1 #> Loses3,
	Loses2 #> Loses3.

solution([[Score1,Wins1,Draws1,Loses1],
	  [Score2,Wins2,Draws2,Loses2],
	  [Score3,Wins3,Draws3,Loses3]]):-
	tournament([[Score1,Wins1,Draws1,Loses1],
		   [Score2,Wins2,Draws2,Loses2],
		   [Score3,Wins3,Draws3,Loses3]]),
	label([Score1, Score2, Score3,
		   Wins1,Wins2,Wins3,
		   Draws1,Draws2,Draws3,
		   Loses1,Loses2,Loses3]).

Calculator with Two Buttons with Prolog

Sat, 14 Nov 2020 00:00:00 +0000

Below my #prolog solution for “Calculator with Two Buttons” proposed by dmcommunity.org challenge Nov 2020

swipl nov2020.pl

?- shortest_path(0,5034,Path), length(Path,Len), findall(Op,member([_,Op,_], Path), Ops).
Path = [[0,+,1],[1,+,2],[2,+,3],[3,+,4],[4,+,5],[5,*,50],[50,*,500],[500,+,501],[501,+,502],[502,+,503],[503,*,5030],[5030,+,5031],[5031,+,5032],[5032,+,5033],[5033,+,5034]],
Len = 15,
Ops = [+,+,+,+,+,*,*,+,+,+,*,+,+,+,+].

Below my nov2020.pl script

:- use_module(library(clpfd)).

shortest_path(From, To, Path):-
	From #>= 0,
	shortest_path(From, To, 0, Path).

shortest_path(From, To, MaxDepth, Path):-
	path(From, To, MaxDepth, Path),! .
shortest_path(From, To, MaxDepth, Path):-
	MaxDepth1 #= MaxDepth + 1,
	shortest_path(From, To, MaxDepth1, Path).

path(From, To, _MaxDepth, [[From, Op, To]]):-
	step(From, Op, To).

path(From, To, MaxDepth, [[From, Op, Mid]|Path]):-
	Mid #> From,
	To  #> Mid,
	MaxDepth #> 0,
	Depth1 #= MaxDepth - 1,
	step(From, Op, Mid),
	path(Mid, To, Depth1, Path).

step(N,*, M):-
	M #= N * 10.
step(N,+,M):-
	M #= N + 1.

Active Directory authentication for PostgreSQL users

Mon, 12 Oct 2020 00:00:00 +0000

It is easy, you just need to add to the configuration file /var/lib/postgresql/data/pg_hba.conf

host all all 0.0.0.0/0 ldap ldapserver="myldapserver" ldapbasedn="OU=USERS,DC=group,DC=redaelli,DC=org" ldapbinddn="CN=matteo,OU=USERS,DC=group,DC=redaelli,DC=org" ldapbindpasswd="MySillyPwd" ldapsearchattribute="sAMAccountName" ldapscheme="ldaps"

And inside your database yu need to create a role for the Active director users and then grant them to the required databases.

LDAP search examples from command line

Mon, 12 Oct 2020 00:00:00 +0000

setup

In these examples I use kerberos authentication but you could also use basic authentication with -D and -w options

For semplicity I put common options into the environment and I have created an alias

export LDAP_OPTS="-h myldap.group.example.com -LLL -Y GSSAPI -b dc=group,dc=example,dc=com  -o ldif-wrap=no "

alias adsearch="/usr/bin/ldapsearch ${LDAP_OPTS}"

Extract locked users

adsearch “(&(objectCategory=Person)(objectClass=User)(lockoutTime>=1))” dn

Expired passwords

date -d “1601/1/1+$(expr 132469210596077795 / 10000000 )Seconds”

“(&(objectclass=user)(objectcategory=person)(!pwdlastset=0)(pwdlastset<={date})(!userAccountControl:1.2.840.113556.1.4.803:=65536))”

Extracting the flat list of members of a group

How to recursively retreiving teh list of members of a group

“(&(objectclass=user)(memberOf:1.2.840.113556.1.4.1941:=CN=Qliksense_Professional,OU=Common Services,DC=redaelli,DC=org))”

Using a GraphQL gateway for backend services (Active Directory, AWS and Qliksense Api samples)

Sun, 11 Oct 2020 00:00:00 +0000

Complex web sites read and write data from/to several backend systems using different interfaces (sql, soap , rest, rpc,..). But it could be simpler and useful to create a single endpoint and interface for all the backends.

With GraphQL the frontend applications get from the backends only the list of fields they need and do not receive the static list of the fields provided by the soap/rest services.

I played with graphql and Walmart lacinia implementing one GraphQL backend for LDAP/Active Directory and one for Qliksense Repository rest api.

When you query an ldap server asking for members of a group, you receive the list of distinguishedNames of the members (see below the values of the field “member”). With GraphQL you can add other fields (see below “memberObjects”) for retreiving more useful info.

I also added other “meta” fields like ip addresses for computer objects

Sample ldap query:

{
	ldap_objects(
		searchdn: "dc=redaelli,dc=org",
		filter: "(sAMAccountName=qliksense_milano)")
	{
		dn
		member
		memberObjects {
			displayName
			sAMAccountName
			mail
		}
	}
}

Sample result:

{
  "data": {
	"ldap_objects": [
	  {
		"dn": "CN=Qliksense_Milano,OU=MngScope,OU=IT,DC=redaelli,DC=org",
		"member": [
		  "CN=user1,OU=Standard GUI Users,DC=redaelli,DC=org",
		  "CN=user2,OU=Standard GUI Users,DC=redaelli,DC=org"],
		"memberObjects": [
		  {
			"displayName": "User1",
			"sAMAccountName": "user1",
			"mail": "name1.lastname1@redaelli.org"
		  },
		  {
			"displayName": "User2",
			"sAMAccountName": "user2",
			"mail": "name2.lastname2@redaelli.org"
		  }]

Sample ldap query extracting computers and IP addresses

{
  ldap_objects(
	system: "group",
	searchdn: "dc=group,dc=redaelli,dc=org",
	filter: "(&(objectClass=computer)(sAMAccountName=dmcAAA*))"
  )
  {
	operatingSystem
	operatingSystemVersion
	dNSHostName
		ipAddresses
  }
}

Result:

{
  "data": {
	"ldap_objects": [
	  {
		"operatingSystem": "Windows Server 2016 Datacenter",
		"operatingSystemVersion": "10.0 (14393)",
		"dNSHostName": "DMCAAAW.group.redaelli.org",
		"ipAddresses": [
		  "20.12.9.4"
		]
	  },

Sample qliksense query:

{
  qliksense_streams(
	system: "myqlikserver"
	filter:"name eq 'Milan'") {
	id
	name
	customProperties {
	  value
	  definition {
		name
	  }
	}}}

Result:

{
  "data": {
	"qliksense_streams": [
	  {
		"id": "10a842ba-cdda-48fa-93f6-z53c92cdad61",
		"name": "Milan",
		"customProperties": [
		  {
			"value": "Qliksense_Milan",
			"definition": {
			  "name": "GroupAccess"
		  }}
	  ]}]}}

The libraries can be found at

Using chrome/firefox programmatically for navigating a (Qliksense) website and taking screenshots

Tue, 08 Sep 2020 00:00:00 +0000

How do you know if the user interface (UI) of a website you’re developing works as it should and the site as a whole delivers the optimal user experience (UX)? Headless browsers give you a fast, lightweight way to automate high-level actions and get an idea of how well your site operates in common scenarios.

Links to Headless browers:

I found the nice clojure library etaoin for using a browser programmatically. Below my code

(ns org.redaelli.qlik-bot-navigation
  (:use [etaoin.api])
  (:require [etaoin.keys :as k])
  (:gen-class))

(defn -main
  "Navigate qliksense pages and then take a screeshot."
  [& args]

  (with-chrome {} ff ;; additional options first, then bind name
	(doto ff
	  (go "https://qliksense.redaelli.org/")
	  ;; form login
	  (fill-multi {:username-input "REDAELLI\\matteo"
				   :password-input "MySillyPassword"})
	  (click {:tag :input
			  :type :submit})

	  (wait-visible {:class "lui-list__item lui-list__action q-li-stream"})

	  ;; enter the first stream
	  (click {:tag :li
			  :class "lui-list__item lui-list__action q-li-stream"})

	  ;; take a screenshot
	  (screenshot (java.io.File. "screenshot.png")))))

Howto quickly extract LDAP attributes for a list of users

Mon, 07 Sep 2020 00:00:00 +0000

I had a list of email addresses and my boss asked me to extract the login names (SamAccountname attribute in Active Directory). I quickly did it with:

  cat mails.txt | while read mail
  do
	echo /usr/bin/curl  -s --proxy \"\" -u \"MYDOMAIN\\myuser:mypassword\" \"ldap://ldapserver:3268/DC=group,DC=mycompany,DC=com?sAMAccountName?sub?(mail=${mail})\"
  done  | bash | grep "sAMAccountName:" | cut -f2 -d":" | sed -e 's/^ /"/' -e 's/$/",/'

Switching from Wordpress to a static website

Mon, 20 Jul 2020 00:00:00 +0000

I migrated my Blog site from Wordpress to a faster static web site using markdown files and Hugo.

Doctor Planning resolved with #Prolog

Wed, 22 Apr 2020 15:36:33 +0000

Below my #prolog solution for “Doctor Planning” proposed by dmcommunity.org challenge April 2020

There should be more constraints like a limit of shifts a week for each doctor… In any case after few seconds I get the first result [[2,3,4],[2,3,4],[2,3,4],[2,3,4],[1,2,3],[1,2,4],[1,2,4]]

It means: Monday sheets: doctor 2 (early), doctor 3 (late) and doctor 4 (night)…

:- use_module(library(clpfd)). /* using swi-prolog */
:- use_module(library(clpz)). /* using scryer-prolog */
/*
  solver for issue
  https://dmcommunity.org/challenge/challenge-apr-2020/
  tested with swi-prolog and scryer-prolog
*/

/*
  constraint 1:
  a doctor can work only a shift a day
*/
constraint1_one_shift_a_day(Doctors):-
	all_different(Doctors).

/*
  constraint 2:
  a doctor should always be available
  for his shift
*/
constraint2_doctor_day_shift(1, 5, _).
constraint2_doctor_day_shift(1, 6, _).
constraint2_doctor_day_shift(1, 7, _).
constraint2_doctor_day_shift(2, _, 1).
constraint2_doctor_day_shift(2, _, 2).
constraint2_doctor_day_shift(3, Day, _):- Day in 1..5.
constraint2_doctor_day_shift(3, Day, Shift):- Day in 6..7, Shift in 1..2.
constraint2_doctor_day_shift(4, _, _).
constraint2_doctor_day_shift(5, _, _).

constraint2_doctor5(Doctors):-
	findall(5, member([_,_,5], Doctors), Turns),
	length(Turns, Tot), Tot #<2.

/*
  constraint 3:
  if a doctor has a night shift,
  they either get the next day off or
  the night shift again
*/
constraint3_two_shifts_rest([[D11,D12,D13],
			     [D21,D22,D23],
			     [D31,D32,D33],
			     [D41,D42,D43],
			     [D51,D52,D53],
			     [D61,D62,D63],
			     [D71,D72,D73]
			    ]):-
	D13 #\= D21,
	D13 #\= D22,
	D23 #\= D31,
	D23 #\= D32,
	D33 #\= D41,
	D33 #\= D42,
	D43 #\= D51,
	D43 #\= D52,
	D53 #\= D61,
	D53 #\= D62,
	D63 #\= D71,
	D63 #\= D72,
	D73 #\= D11,
	D73 #\= D12.

/*
  constaint 4:
  both days in the weekend on none
*/
constraint4_both_saturday_sunday([_,
				  _,
				  _,
				  _,
				  _,
				  [D61,D62,D63],
				  [D61,D62,D63]
				 ]).
constraint4_both_saturday_sunday([_,
				  _,
				  _,
				  _,
				  _,
				  [D61,D62,D63],
				  [D62,D61,D63]
				 ]).

overall_constraints(Doctors):-
	constraint3_two_shifts_rest(Doctors),
	constraint4_both_saturday_sunday(Doctors),
	constraint2_doctor5(Doctors).

solve_one_day(Day, Doctors):-
	length(Doctors,3),
	Doctors ins 1..5,
	constraint1_one_shift_a_day(Doctors),
	Doctors = [Doctor1, Doctor2, Doctor3],
	/* constraint 2 */
	constraint2_doctor_day_shift(Doctor1, Day, 1),
	constraint2_doctor_day_shift(Doctor2, Day, 2),
	constraint2_doctor_day_shift(Doctor3, Day, 3).

solve_day_by_day([], _Doctors).
solve_day_by_day([Day|Days], [DayDoctors|Doctors]):-
	solve_one_day(Day, DayDoctors),
	solve_day_by_day(Days, Doctors).

solve(Doctors):-
	solve_day_by_day([1,2,3,4,5,6,7], Doctors),
	overall_constraints(Doctors).

Comments:

Gergö Barany - Apr 6, 2020

Nice solution! I was wondering how to approach this with CLP(FD), it did not seem “natural” to me. Here is my solution using a more “symbolic” approach, not using numerical constraints but with DCGs and aggressive use of dif/2: % A day is a list of three Doctor-Day-Shift terms. Only one shift per day % for each doctor. day(Day, EarlyDoc, LateDoc, NightDoc) –> [EarlyDoc-Day-early, LateDoc-Day-late, NightDoc-Day-night], { dif(EarlyDoc, LateDoc), dif(EarlyDoc, NightDoc), dif(LateDoc, NightDoc) }. % A week is a list of Doctor-Day-Shift terms. week –> day(sun, EarlySun, LateSun, NightSun), % Whoever works this night shift does not work the early or late shift % the next day. { dif(NightSun, EarlyMon), dif(NightSun, LateMon) }, day(mon, EarlyMon, LateMon, NightMon), { dif(NightMon, EarlyTue), dif(NightMon, LateTue) }, day(tue, EarlyTue, LateTue, NightTue), { dif(NightTue, EarlyWed), dif(NightTue, LateWed) }, day(wed, EarlyWed, LateWed, NightWed), { dif(NightWed, EarlyThu), dif(NightWed, LateThu) }, day(thu, EarlyThu, LateThu, NightThu), { dif(NightThu, EarlyFri), dif(NightThu, LateFri) }, day(fri, EarlyFri, LateFri, NightFri), { dif(NightFri, EarlySat), dif(NightFri, LateSat) }, day(sat, EarlySat, LateSat, NightSat), { dif(NightSat, EarlySun), dif(NightSat, LateSun) }, % The week starts and ends with a Sunday to ensure all constraints wrap % around correctly. In this formulation, both Sundays have the same % schedule, but this could be relaxed. day(sun, EarlySun, LateSun, NightSun). % Just the general constraints, no doctor-specific ones yet. schedule_(Schedule) :- phrase(week, Schedule). day(mon). day(tue). day(wed). day(thu). day(fri). day(sat). day(sun). weekend(sat). weekend(sun). shift(early). shift(late). shift(night). % Doctor-specific availability rules. doctor_available(fleming-Day-Shift) :- shift(Shift), member(Day, [fri, sat, sun]). doctor_available(freud-Day-Shift) :- member(Shift, [early, late]), day(Day). doctor_available(heimlich-Day-Shift) :- day(Day), shift(Shift), \+ ( weekend(Day), Shift = night ). doctor_available(eustachi-Day-Shift) :- day(Day), shift(Shift). doctor_available(golgi-Day-Shift) :- day(Day), shift(Shift). % but at max 2 night shifts, see golgi_rule/1 % A day with actual doctors assigned. scheduled_day(Day) –> [EarlyDoc-Day-early, LateDoc-Day-late, NightDoc-Day-night], { doctor_available(EarlyDoc-Day-early), doctor_available(LateDoc-Day-late), doctor_available(NightDoc-Day-night) }. % A week with doctors assigned. scheduled_week –> scheduled_day(sun), scheduled_day(mon), scheduled_day(tue), scheduled_day(wed), scheduled_day(thu), scheduled_day(fri), scheduled_day(sat), scheduled_day(sun). weekend_rule(Schedule) :- Weekend = [SatEarlyDoc-sat-early, SatLateDoc-sat-late, SatNightDoc-sat-night, SunEarlyDoc-sun-early, SunLateDoc-sun-late, SunNightDoc-sun-night], append(_Prefix, Weekend, Schedule), % The doctors working the Saturday shifts must have Sunday shifts as % well. SunDocs = [SunEarlyDoc, SunLateDoc, SunNightDoc], member(SatEarlyDoc, SunDocs), member(SatLateDoc, SunDocs), member(SatNightDoc, SunDocs), % There is no doctor working a Saturday and not Sunday, or vice versa. % This is redundant, it is implied by the other constraints. \+ ( member(Doctor-sat-_, Schedule), \+ member(Doctor-sun-_, Schedule) ), \+ ( member(Doctor-sun-_, Schedule), \+ member(Doctor-sat-_, Schedule) ). golgi_rule(Schedule) :- bagof(Day, Shift^member(golgi-Day-Shift, Schedule), GolgiNights), length(GolgiNights, N), N =< 2. % Main predicate: A fully scheduled week with all constraints applied. schedule(Schedule) :- schedule_(Schedule), phrase(scheduled_week, Schedule), golgi_rule(Schedule), weekend_rule(Schedule). Tested with SWI-Prolog 7.2.3. Example solution: ?- time(schedule(S)), maplist(writeln, S). % 26,781 inferences, 0.003 CPU in 0.003 seconds (100% CPU, 7957758 Lips) fleming-sun-early freud-sun-late eustachi-sun-night freud-mon-early heimlich-mon-late eustachi-mon-night freud-tue-early heimlich-tue-late eustachi-tue-night freud-wed-early heimlich-wed-late eustachi-wed-night freud-thu-early heimlich-thu-late eustachi-thu-night fleming-fri-early freud-fri-late golgi-fri-night fleming-sat-early freud-sat-late eustachi-sat-night fleming-sun-early freud-sun-late eustachi-sun-night S = [fleming-sun-early, freud-sun-late, eustachi-sun-night, freud-mon-early, heimlich-mon-late, eustachi-mon-night, freud-tue-early, … - … - late, … - …|…] . If I’m not mistaken, this is equivalent to your sample solution except for having Golgi on Friday night instead of Heimlich.

Imperative (not functional) languages are so boring. Long life to logical and declarative programming like Prolog

Sat, 18 Apr 2020 19:21:00 +0000

Below a simple example of using a a declarative language (Prolog) for finding all solutions for a trick game…

?- use_module(library(clpfd)).
?- X in 1..9, Y #= (X * 3 + 3) * 3, Z #= (Y rem 10) + (Y div 10), setof(Z, X^Y^label([X,Y,Z]), Sol).

Sol = [9],

My Networking Survival Kit

Sun, 15 Mar 2020 21:44:38 +0000

In this small tutorial I’ll speak about tunneling, ssh port forwarding, socks, pac files, Sshuttle

I’ve been using Linux since 1995 but I have never been interested a lot in networking. In these many days of smart working (due to Covid-19) I have found some useful tricks to connect to remote systems that are not directly reachable from my lan/vpn

Case 1 (port forwarding): I wanted to connect to targethost.redaelli.org at tcp port 10000 but I was not able to reach it directly but only through an other host (tunnelhost.redaelli.org). With the following command I was able to reach the target host connecting to localhost:10000

ssh -NL 10000:targethost.redaelli.org:10000 r@tunnelhost.redaelli.org

Case 2 (many port forwarding): I added in the file $HOME/.ssh/config

Host tunnelhost
User matteo
Hostname tunnelhost.redaelli.org
LocalForward 10000 192.168.20.152:10000
LocalForward 10001 192.168.40.123:10000
LocalForward 10002 192.168.60.112:10000

And after running “ssh -N tunnelhost” I was able to reach the target systems through localhost:10000, localhost:10001 and localhost:10002

Case 3 (socks5): connecting to many remote hosts using their hostnames (and not localhost). I started a socks server with

ssh -D 9999 -q -C -N matteo@tunnelhost.redaelli.org

An then I configured it in network settings in Firefox.

A further improvement was to create a “pac” file and set it in the network settings in Firefox. My pac file was

function FindProxyForURL(url, host) {
    var useSocks = ["remotehost.redaelli.org", 
                    "remotehost2.redaelli.org"];

    for (var i= 0; i < useSocks.length; i++) {
	if (shExpMatch(host, useSocks[i])) {
	    return "SOCKS localhost:9999 ; DIRECT";
	}
    }
    if (isInNet(dnsResolve(host), "192.168.20.0", "255.255.255.0") ||
        isInNet(dnsResolve(host), "192.168.40.0", "255.255.255.0") ||
        isInNet(dnsResolve(host), "192.168.60.0", "255.255.255.0")
       ) {
	    return "SOCKS localhost:9999 ; DIRECT";
    }
    return "DIRECT";
}

Case 4 (sshuttle): connecting to many remote hosts like native connections (without socks and pac files).

“Sshuttle is not exactly a VPN, and not exactly port forwarding. It’s kind of both, and kind of neither.”

I installed sshuttle with “apt-get install sshuttle” on my debian laptop and then I ran

sshuttle -r matteo@tunnelhost.redaelli.org 192.168.20.0/24 192.168.40.0/24 192.168.60.0/24

For sure there are many other more powerful and better solutions. But for the moment these are the one I have used.

How to backup and restore Glue data catalog

Fri, 21 Feb 2020 14:46:17 +0000

How to recover a wrongly deleted glue table? You should have scheduled a periodic backup of Glue data catalog with

aws glue get-tables --database-name mydb > glue-mydb.json

And recreate your table with the command

aws glue create-table --cli-input-json '{...}'

But the json format of aws glue get-tables is quite different from the json format of aws create-table. For the conversion you can use a simple python script like the following one

import json, sys

def dict_convert(dict):
    DatabaseName = dict['DatabaseName']
    del dict['DatabaseName']
    del dict['CreateTime']
    del dict['UpdateTime']
    del dict['IsRegisteredWithLakeFormation']
    result = {
        'DatabaseName': DatabaseName,
        'TableInput': dict
    }
    return result

data = json.load(sys.stdin)

for t in data["TableList"]:
    print(json.dumps(dict_convert(t), default=str))

N queens problem in Prolog

Fri, 10 Jan 2020 17:59:40 +0000

Below the N Queens problem solved in Prolog and Constraint Logic Programming over Finite Domains library

:- use_module(library(clpfd)).

n_queens(N, Queens) :-
        length(Queens, N),
        Queens ins 1..N,
	all_different(Queens), %% the queens must be in different columns
        different_diagonals(Queens).

different_diagonals([]).
different_diagonals([Q|Queens]) :- different_diagonals(Queens, Q, 1), different_diagonals(Queens).

different_diagonals([], _, _).
different_diagonals([Q|Queens], Q0, Distance) :-
        abs(Q0 - Q) #\= Distance,
        NewDistance #= Distance + 1,
        different_diagonals(Queens, Q0, NewDistance).

/* 
   Queens is the list of columns of the queens: the corresponding rows are the position of the elements/columns in the list Queens

   Examples:

   ?- n_queens(8, Queens), labeling([ff], Queens).
   %@ Queens = [1, 5, 8, 6, 3, 7, 2, 4] ;
   %@ Queens = [1, 6, 8, 3, 7, 4, 2, 5] .

   The result chess cells are, for instance, [1,1], [2,5], [3,8], [4,6], [5,3], [6,7], [7,2], [8,4]

   Suggestion from https://www.swi-prolog.org/pldoc/man?section=clpfd-n-queens
 */

Prolog and Constraint Logic Programming over Finite Domains

Mon, 06 Jan 2020 22:11:34 +0000

I like Prolog and in these days I have studied the library CLP(FD).

For instance it is easy to write a simple code for solving “How many men and horses have 8 heads and 20 feet?”. You write the rules and contraints and Prolog will find the solution for you

men_and_horses(Men, Horses):-
	Men in 0..10,
	Horses in 0..10,
	Men + Horses #= 8, %% heads must be 8
	Men * 2 + Horses * 4 #= 20. %% feet must be 20

?- men_and_horses(Men, Horses).
 Men = 6,
 Horses = 2.

“clp(fd) is useful for solving a wide variety of find values for these variables problems. Here are some broad categories of problems that clp(fd) can address:

Scheduling problems, like, when should we do what work in this factory to make these products?
Optimization problems, like which mix of products should we make in this factory to maximize profits?
Satisficing problems, like finding an arrangement of rooms in a hospital that meets criteria like having the operating theater near the recovery rooms, or finding a set of vacation plans the whole family can agree on.
Sequence problems, like finding a travel itinerary that gets us to our destination.
Labeling problems, like Sudoku or Cryptarithm puzzles
…. " [See what_is_clp_fd_good_for]

Running Talend Remote Engine in a docker container

Tue, 05 Nov 2019 21:42:39 +0000

I was not able to find a Dockerfile for running Talend Remote Engine in a container. So I tried to build a new one. It is a working in progress: do you have any suggestions?

TODO / Next steps:

registering the engine using Talend API
running the engine with a unix user “talend”

FROM centos:7
# centos is the recommended linux distribution
MAINTAINER  Matteo Redaelli 

# Build and run this image with:
# - docker build -t  talend/remote_engine:2.7.0
# - docker run -d --name talend_remote_engine talend/remote_engine:2.7.0 run

# Set environment variables.
ENV TALEND_HOME /opt/talend
ENV TALEND_ENGINE_HOME $TALEND_HOME/remote_engine
ENV HOME $TALEND_HOME
ENV JAVA_VERSION=java-1.8.0-openjdk
ENV JAVA_HOME /usr/lib/jvm/$JAVA_VERSION

# Installing java
RUN yum update -y && \
   yum install -y $JAVA_VERSION ${JAVA_VERSION}-devel && \
   rm -rf /var/cache/yum

# Define working directory.
WORKDIR $TALEND_HOME

## remember to update config files before creating the image
## - Talend-RemoteEngine-*/etc/preauthorized.key.cfg with the engine key, name and description
## - Talend-RemoteEngine-*/etc/system.properties with proxy settings (if needed)
COPY Talend-RemoteEngine-V2.7.0 $TALEND_ENGINE_HOME

#RUN mkdir $HOME/.m2
#COPY settings.xml $HOME/.m2/settings.xml

# Define default command.
# See trun source for options: you shuld use "run"
ENTRYPOINT ["/opt/talend/remote_engine/bin/trun"]

Comments:

Thibaut Gourdel - Nov 3, 2019

Hello Matteo, thanks for sharing this. It’s a great start to containerize your Talend Remote Engine! What you could add is an entrypoint file to configure your remote engine at runtime (instead of build time). What I mean is to pass the preauthorizedkey and other configuration variables via environment variable to the container so that you build your remote engine once and you can run it with different parameters. You can achieve that by having a shell script as the entrypoint, configure the config file via this shell script based on environment variables and then start the remote engine. At the end it would look like this: docker run -d -e PREAUTHORIZEDKEY=****** -e WEB_LISTENING_PORT=8043 talend/remote_engine:2.7.0 run Thibaut

Emacs package sql-sqlline and python package sqlu

Mon, 04 Nov 2019 15:27:07 +0000

Few days ago I published the following two opensource packages:

sql-sqlline: an emacs package that extends sql package for quering less known databases (aws redshift, aws athena, ..)
sqlu: a python package for extracting tables from sql select statements

Comments:

buy Andriol - Mar 2, 2020

I don’t know if it’s just me or if perhaps everybody else experiencing issues with yor site. It appears as though some of the written text on your posts are running off the screen. Can somebody else please comment and let me know if this is happening to them as well? This could be a issue with my internet browser because I’ve had this happen before. Many thanks buy Andriol

Using Apache Camel from Groovy

Thu, 03 Oct 2019 13:22:27 +0000

Apache Camel is an open source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data.

Apache Groovy is a Java-syntax-compatible object-oriented programming language for the Java platform. It is both a static and dynamic language with features similar to those of Python, Ruby, and Smalltalk. It can be used as both a programming language and a scripting language for the Java Platform, is compiled to Java virtual machine (JVM) bytecode, and interoperates seamlessly with other Java code and libraries. Groovy uses a curly-bracket syntax similar to Java’s. Groovy supports closures, multiline strings, and expressions embedded in strings. Much of Groovy’s power lies in its AST transformations, triggered through annotations. [Wikipedia]

Create a file camel-test.groovy like the following

 @Grab('org.apache.camel:camel-core:2.21.5')
 @Grab('javax.xml.bind:jaxb-api:2.3.0')
 @Grab('org.slf4j:slf4j-simple:1.7.21')
 @Grab('javax.activation:activation:1.1.1')

 import org.apache.camel.\*
 import org.apache.camel.impl.\*
 import org.apache.camel.builder.\*
 def camelContext = new DefaultCamelContext()
 camelContext.addRoutes(new RouteBuilder() {
	 def void configure() {
		 from("timer://jdkTimer?period=3000")
			 .to("log://camelLogger?level=INFO")
	 }
 })
 camelContext.start()
 addShutdownHook{ camelContext.stop() }
 synchronized(this){ this.wait() }

Test it with

JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 groovy camel-test.groovy

Using Terraform for managining Amazon Web Service infrastructure

Mon, 30 Sep 2019 12:18:23 +0000

In the last days I tested Terraform (Use Infrastructure as Code to provision and manage any cloud, infrastructure, or service) for managing some resources in a AWS cloud ebvironemnt.

In this sample I’ll create and schedule a lambda function

Create a file “variables.tf” with the content:

variable "aws_region" {default = "eu-west-1"}
variable "aws_profile" {default = ""}
variable "project" {default = "my_project"}

variable "vpc" {default= "XXXXX"}
variable "subnets" {default= "XXXX"}
variable "aws_account" {default= "XXX"}
variable "security_groups" {default= "XXXX"}
#
variable "db_redshift_host" {default= ""}
variable "db_redshift_port" {default= ""}
variable "db_redshift_name" {default= ""}
variable "db_redshift_username" {default= ""}
variable "db_redshift_password" {default= ""}

Create a file lambda.tf as follow:

provider "aws" {
  region  = "${var.aws_region}"
  profile = "${var.aws_profile}"
}
# ############################################################################
# CLOUDWATCH
# ############################################################################
resource "aws_cloudwatch_log_group" "log_group" {
  name              = "/aws/lambda/${var.project}"
  retention_in_days = 14
}

# ############################################################################
# CLOUDWATCH rules
# ############################################################################
resource "aws_cloudwatch_event_rule" "rule" {
  name        = "${var.project}-rule"
  description = "scheduler for ${var.project}"
  schedule_expression = "cron(0 10 * * ? *)"
}
resource "aws_cloudwatch_event_target" "trigger_lambda" {
  rule  = "${aws_cloudwatch_event_rule.rule.name}"
  arn   = "${aws_lambda_function.lambda.arn}"
}

# ############################################################################
# iam
# ############################################################################
resource "aws_iam_role" "role" {
  name = "${var.project}_role"
  #assume_role_policy = "${file("assumerolepolicy.json")}"
  assume_role_policy = <
{
"Version": "2012-10-17",
"Statement": [
  {
    "Action": "sts:AssumeRole",
    "Principal": {
      "Service": "lambda.amazonaws.com"
    },
    "Effect": "Allow",
    "Sid": ""
  }
]
}
EOF
}

resource "aws_iam_policy" "logging" {
  name = "${var.project}_logging"
  path = "/"
  description = "${var.project} IAM policy for logging"

  policy = <
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*",
      "Effect": "Allow"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "logging" {
  role = "${aws_iam_role.role.name}"
  policy_arn = "${aws_iam_policy.logging.arn}"
}

resource "aws_iam_role_policy_attachment" "policy_attachment_vpc" {
  #name       = "${var.project}_attachment_vpc"
  role       = "${aws_iam_role.role.name}"
  policy_arn = "arn:aws:iam::aws:policy/AmazonVPCFullAccess"
}

resource "aws_iam_role_policy_attachment" "policy_attachment_rds" {
  role       = "${aws_iam_role.role.name}"
  policy_arn = "arn:aws:iam::aws:policy/AmazonRDSReadOnlyAccess"
}

resource "aws_iam_role_policy_attachment" "policy_attachment_redshift" {
  role       = "${aws_iam_role.role.name}"
  policy_arn = "arn:aws:iam::aws:policy/AmazonRedshiftReadOnlyAccess"
}

# ###############################################
# lambda_action
# ###############################################

resource "aws_lambda_function" "lambda" {
  function_name = "${var.project}_lambda"
  depends_on    = ["aws_iam_role_policy_attachment.logging", "aws_cloudwatch_log_group.log_group"]
  filename      = "lambda.zip"
  role          = "${aws_iam_role.role.arn}"
  handler       = "lambda_function.lambda_handler"
  source_code_hash = "${filebase64sha256("lambda.zip")}"
  runtime = "python3.7"
  timeout          = "30"
  memory_size      = 256
  publish          = true
  vpc_config {
    subnet_ids = "${var.subnets}"
    security_group_ids = "${var.security_groups}"
  }

  environment {
    variables = {
      db_redshift_host= "${var.db_redshift_host}"
      db_redshift_port= var.db_redshift_port
      db_redshift_name= "${var.db_redshift_name}"
      db_redshift_username= "${var.db_redshift_username}"
      db_redshift_password= "${var.db_redshift_password}"
    }
  }
}

Now you can run

terraform init 
terraform  plan -var aws\_profile=myprofile
terraform  apply -var aws\_profile=myprofile 
terraform  destroy -var aws\_profile=myprofile

Scheduling AWS EMR clusters resize

Mon, 22 Jul 2019 21:16:37 +0000

Below a sample of howto schedule an Amzon Elastic MapReduce (EMR) cluster resize. It is useful if you have a cluster that is less used during the nights or in the weekends

I used a lambda function triggered by a Cloudwatch rule. Here is my python lambda function

import boto3, json

MIN=1
MAX=10

def lambda_handler(event, context):
    region = event["region"]
    ClusterId = event["ClusterId"]
    InstanceGroupId = event["InstanceGroupId"]
    InstanceCount = int(event['InstanceCount'])
    
    if InstanceCount >= MIN and InstanceCount <= MAX:
        client = boto3.client('emr', region_name=region)
        response = client.modify_instance_groups(
            ClusterId=ClusterId,
            InstanceGroups= [{
                "InstanceGroupId": InstanceGroupId,
                "InstanceCount": InstanceCount
            }])
        return response
    else:
        msg = "EMR cluster id %s (%s): InstanceCount=%d is NOT allowed [%d,%d]" % (ClusterId, region, InstanceGroupId, InstanceCount, MIN,MAX)
        return {"response": "ko", "message": msg}

Below the CloudWatch rule where the input event is a constant json object like

{"region": "eu-west-1","ClusterId": "j-dsds","InstanceGroupId": "ig-sdsd","InstanceCount": 8}

Exporting database tables to csv files with Apache Camel

Fri, 24 May 2019 07:12:48 +0000

Below the interested part of code using spring xml

 
		 
		  
		  
		  
  




	
	  
	  
		0
	  
	  
	

	
	  
	  
		table1,table2,table3
	  
	  
		
		
		  ${body}
		
		
	  
	

	
	  
		
		
	

	
	  
	  

	  
				  "SELECT * from " + request.headers.get('tablename')
	]]>
		
	  
	  
	  
	

	
	  
			
	  
		
		  request.headers.get('tablename').replace(".", "_") + "/" + request.headers.get('tablename') + ".csv"
		
	  

	  

	  
		
		  
		  
	request.headers.get('CamelJdbcColumnNames').join(";") + "\n"

Active Directory client with powershell

Fri, 19 Apr 2019 07:37:24 +0000

Add AD users from csv to group using powershell

A sample script for adding users (taken from a csv file) to an Active Directory group

$GroupName = "Qliksense_SI_Techedge"
$Users =  "e:\scripts\users.csv"

Import-module ActiveDirectory

$dc = Get-ADDomainController -DomainName mydomain.redaelli.org -Discover -NextClosestSite
$server = $dc.HostName[0]

get-content $Users | ForEach-Object {
  Get-ADUser -Server $server -LDAPFilter "(mail=$_)" } |
  Select-Object -ExpandProperty sAMAccountName |  ForEach-Object { Add-ADGroupMember -Server $server -Identity $GroupName -Member $_ }

AWS Lake Formation: the new Datalake solution proposed by Amazon

Thu, 29 Nov 2018 21:04:11 +0000

AWS Lake Formation is a service that makes it easy to set up a secure data lake in days. A data lake is a centralized, curated, and secured repository that stores all your data, both in its original form and prepared for analysis. A data lake enables you to break down data silos and combine different types of analytics to gain insights and guide better business decisions.

However, setting up and managing data lakes today involves a lot of manual, complicated, and time-consuming tasks. This work includes loading data from diverse sources, monitoring those data flows, setting up partitions, turning on encryption and managing keys, defining transformation jobs and monitoring their operation, re-organizing data into a columnar format, configuring access control settings, deduplicating redundant data, matching linked records, granting access to data sets, and auditing access over time.

Creating a data lake with Lake Formation is as simple as defining where your data resides and what data access and security policies you want to apply. Lake Formation then collects and catalogs data from databases and object storage, moves the data into your new Amazon S3 data lake, cleans and classifies data using machine learning algorithms, and secures access to your sensitive data. Your users can then access a centralized catalog of data which describes available data sets and their appropriate usage. Your users then leverage these data sets with their choice of analytics and machine learning services, like Amazon EMR for Apache Spark, Amazon Redshift, Amazon Athena, Amazon Sagemaker, and Amazon QuickSight. [aws.amazon.com]

Lake Formation automatically configures underlying AWS services, including S3, AWS Glue, AWS IAM, AWS KMS, Amazon Athena, Amazon Redshift, and Amazon EMR for Apache Spark, to ensure compliance with your defined policies. If you’ve set up transformation jobs spanning AWS services, Lake Formation configures the flows, centralizes their orchestration, and lets you monitor the execution of your jobs. With Lake Formation, you can configure and manage your data lake without manually integrating multiple underlying AWS services

Sources:

Building a Cloud-Agnostic Serverless infrastructure with Apache OpenWhisk

Thu, 29 Nov 2018 20:29:02 +0000

“Apache OpenWhisk (Incubating) is an open source, distributed Serverless platform that executes functions (fx) in response to events at any scale. OpenWhisk manages the infrastructure, servers and scaling using Docker containers so you can focus on building amazing and efficient applications…

DEPLOY Anywhere: Since Apache OpenWhisk builds its components using containers it easily supports many deployment options both locally and within Cloud infrastructures. Options include many of today’s popular Container frameworks such as Kubernetes, Mesos and Compose

ANY LANGUAGES: Work with what you know and love. OpenWhisk supports a growing list of your favorite languages such as NodeJS, Swift, Java, Go, Scala, Python, PHP and Ruby.

If you need languages or libraries the current “out-of-the-box” runtimes do not support, you can create and customize your own executables as Zip Actions which run on the Docker runtime by using the Docker SDK. " [openwhisk.apache.org]

Building a Cloud-Agnostic Serverless infrastructure with Knative

Fri, 23 Nov 2018 08:16:05 +0000

KNATIVE is Kubernetes-based platform to build, deploy, and manage modern serverless workloads

“Knative provides a set of middleware components that are essential to build modern, source-centric, and container-based applications that can run anywhere: on premises, in the cloud, or even in a third-party data center. Knative components are built on Kubernetes and codify the best practices shared by successful real-world Kubernetes-based frameworks. It enables developers to focus just on writing interesting code, without worrying about the “boring but difficult” parts of building, deploying, and managing an application.” [https://cloud.google.com/knative/]

“Knative has been developed by Google in close partnership with Pivotal, IBM, Red Hat, and SAP.” [infoq.com]

Querying public knowledge graph databases

Sat, 18 Aug 2018 19:54:22 +0000

You can query public knowledge graph databases (like wikidata.org and dbpedia.org) using SPARQL. For instance for extracting all “known” programming languages, you can use the query

SELECT ?item ?itemLabel WHERE {
 ?item wdt:P31 wd:Q9143.
 SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO\_LANGUAGE],en". }
}
LIMIT 1000

There are also SPARQL clients for most of programming languages. With (swi) prolog you can easily run

\[library(semweb/sparql\_client)\].
sparql\_query('SELECT ?item ?itemLabel WHERE {?item wdt:P31 wd:Q9143. SERVICE wikibase:label { bd:serviceParam wikibase:language "\[AUTO\_LANGUAGE\],en". }} LIMIT 1000', Row, \[ scheme(https),host('query.wikidata.org'), path('/sparql')\]).

Adding Active Directory authentication to RStudio Server Open Source edition

Wed, 28 Mar 2018 20:43:01 +0000

Have you installed the statistical R language and RStudio Server in a corporare environment and you would like to authenticate users using Active Directory instead of local unix users and passwords? It is easy, you just need to install libpam-krb5… ;-) Below a sample Dockerfile if you want to install RStudio Server in a Docker container… FROM rocker/rstudio RUN apt-get update ADD krb5.conf /etc RUN apt-get install -y krb5-config libpam-krb5 RUN apt-get install -y openjdk-8-jdk r-cran-rjava #setup R configs RUN echo “r <- getOption(‘repos’); r[‘CRAN’] <- ‘http://cran.us.r-project.org’; options(repos = r);” > ~/.Rprofile #INSTALL LIBAIO1 & UNZIP (NEEDED FOR STRONG-ORACLE) RUN apt-get update \ && apt-get install -y libaio1 \ && apt-get install -y build-essential \ && apt-get install -y unzip \ && apt-get install -y curl \ && rm -rf /var/lib/apt/lists/* #ADD ORACLE INSTANT CLIENT RUN mkdir -p opt/oracle ADD ./oracle/linux/ . RUN unzip instantclient-basic-linux.x64-12.2.0.1.0.zip -d /opt/oracle \ && unzip instantclient-sdk-linux.x64-12.2.0.1.0.zip -d /opt/oracle \ && unzip instantclient-odbc-linux.x64-12.2.0.1.0-2.zip -d /opt/oracle \ && unzip instantclient-jdbc-linux.x64-12.2.0.1.0.zip -d /opt/oracle \ && mv /opt/oracle/instantclient_12_2 /opt/oracle/instantclient \ && ln -s /opt/oracle/instantclient/libclntsh.so.12.1 /opt/oracle/instantclient/libclntsh.so \ && ln -s /opt/oracle/instantclient/libocci.so.12.1 /opt/oracle/instantclient/libocci.so \ && rm instantclient-*zip #ENV ORACLE_HOME="/opt/oracle/instantclient" ENV LD_LIBRARY_PATH="/opt/oracle/instantclient" #ENV OCI_HOME="/opt/oracle/instantclient" ENV OCI_LIB="/opt/oracle/instantclient" ENV OCI_INCLUDE_DIR="/opt/oracle/instantclient/sdk/include" RUN echo ‘/opt/oracle/instantclient/’ | tee -a /etc/ld.so.conf.d/oracle_instant_client.conf && ldconfig RUN Rscript -e “install.packages(‘yhatr’)” RUN Rscript -e “install.packages(‘ggplot2’)” RUN Rscript -e “install.packages(‘plyr’)” RUN Rscript -e “install.packages(‘reshape2’)” RUN Rscript -e “install.packages(‘forecast’)” RUN Rscript -e “install.packages(‘stringr’)” RUN Rscript -e “install.packages(’lubridate’)” RUN Rscript -e “install.packages(‘randomForest’)” RUN Rscript -e “install.packages(‘rpart’)” RUN Rscript -e “install.packages(‘RJDBC’)” RUN Rscript -e “install.packages(‘ROracle’)” RUN adduser -u 2000 –disabled-password –gecos "" matteo RUN adduser -u 2001 –disabled-password –gecos "" marco

Google Assistent in italiano per comandarele luci Xiaomi

Sat, 17 Mar 2018 21:33:46 +0000

Finalmente posso comandare le luci (xiaomi) di casa mia in italiano con Google Assistant! Per esempio funzionano le frasi

accendi le luci
spegni le luci
luce rossa
luce banca calda
…

Latest packages for Elixir Haskell Lua dotNet NodeJs Php Python R Ruby Scala

Thu, 10 Aug 2017 08:45:20 +0000

The site packages.matteoredaelli.eu collects the last updated packages for most of programming languages (at the moment Elixir Haskell Lua dotNet NodeJs Php Python R Ruby Scala)

Prolog for theorem proving, expert systems, type inference systems, and automated planning...

Thu, 25 May 2017 09:46:31 +0000

In the name of the father of Prolog (Alain_Colmerauer who died few days ago), I’ll show how to use Prolog for solving a common business problem: finding the paths in a graph between two nodes.. “Prolog is declarative programming language: the program logic is expressed in terms of relations, represented as facts and rules. A computation is initiated by running a query over these relations”. [Wikipiedia]. “Prolog is a general-purpose logic programming language associated with artificial intelligence and computational linguistics [..] The language has been used for theorem proving, expert systems, type inference systems, and automated planning, as well as its original intended field of use, natural language processing.[Wikipiedia]. You tell Prolog the facts and rules of your game and it will find the solution ;-) In this tutorial my graph is the network of underground/train stations of Milan. The facts are like

station('Affori centro', m3).
station('Affori FN', m3).
station('Affori', s2).
station('Affori', s4).
station('Airuno', s8).
station('Albairate - Vermezzo', s9).
station('Albate Camerlata', s11).
station('Albizzate', s5).
station('Amendola Fiera', m1).
station('Arcore', s8).
station('Assago Milanofiori Forum', m2).
station('Assago Milanofiori Nord', m2).


edge('Villapizzone', 'Lancetti', s5).
edge('Villapizzone', 'Lancetti', s6).
edge('Villa Pompea', 'Gorgonzola', m2).
edge('Villa Raverio', 'Carate-Calò', s7).
edge('Villasanta', 'Monza Sobborghi', s7).
edge('Villa S. Giovanni', 'Precotto', m1).
edge('Vimodrone', 'Cascina Burrona', m2).
edge('Vittuone', 'Pregnana Milanese', s6).
edge('Wagner', 'De Angeli', m1).
edge('Zara', 'Isola', m5).
edge('Zara', 'Sondrio', m3).
```The rules are like```
adiacent(\[X,L1\], \[Y,L1\]) :- edge(X,Y, L1) ; edge(Y, X, L1).

change(L1,L2, X) :-
 station(X,L1),
 station(X,L2),
 not(L1 == L2).

same_line_path(Node, Node, _, [Node]). % rule 1
same_line_path(Start, Finish, Visited, [Start | Path]) :- % rule 2
 adiacent(Start, X),
 not(member(X, Visited)),
 same_line_path(X, Finish, [X | Visited], Path).

one_change_line_path([Start,L1], [End,L2], Visited, Path):-
 station(Start,L1),
 station(End,L2),
 change(L1,L2, X),
 same_line_path([Start,L1], [X,L1], [[Start,L1]|Visited], Path1),
 same_line_path([X,L2], [End,L2], [[X,L2]|Visited], Path2),
 append(Path1, Path2, Path).

You can find the source code of the Prolog webservice at https://github.com/matteoredaelli/metropolitana-milano

howto extract sentences from an ebook/doc/pdf/text file

Wed, 05 Apr 2017 20:43:35 +0000

You can easily extract sentences froma epub/pdf/office/text file thanks to the opensource projects [Apache Tika](http://Apache Tika) and Apache OpenNlp with a command like``` java -jar tika-app-1.14.jar -t ebook.epub | opennlp SentenceDetector it-sent.bi

Deploy tomcat applications in Docker containers

Tue, 14 Mar 2017 16:37:34 +0000

Deploying you applications in containers, you are sure that they are easily portable and scalable… Here a sample of deploying a .war application using a Docker container Create a Dockerfile like``` FROM tomcat:8-jre8

MAINTAINER “Matteo matteo.redaelli@gmail.com”

ADD server.xml /usr/local/tomcat/conf/ ADD tomcat-users.xml /usr/local/tomcat/conf/ ADD ojdbc6.jar /usr/local/tomcat/lib/ ADD bips.war /usr/local/tomcat/webapps/ Build a docker image docker build . -t myapp Run one or more docker images of your appplication with docker run –restart=unless-stopped –name myapp1 -p 8080:8080 -d myapp docker run –restart=unless-stopped –name myapp2 -p 8081:8080 -d myapp It is better to redirect tomcat logs to stdout: in this way you can see them with docker logs myapp

Continuous integration and continuous delivery with Jenkins

Fri, 10 Mar 2017 14:25:45 +0000

In this post I’ll show how to use the opensource tool #Jenkins, “the leading #opensource automation server, Jenkins provides hundreds of plugins to support building, deploying and automating any project”. I’ll create a simple pipeline that executes remote tasks via ssh. It could be used for continuous integration and continuous delivery for Oracle OBIEE Systems

Install (in a docker container)

docker run -p 8080:8080 -p 50000:50000 -v /home/oracle/docker\_shares/jenkins:/var/jenkins\_home -d jenkins

Configure credentials

Configure remote nodes

Jenkins -> Manage Jenkins -> Manage nodes -> Add node

Configure Pipeline Jenkins -> New Item -> Pipeline See https://gist.github.com/matteoredaelli/8d306d79e547f3fdfd5d1c467373f8e0

Log analysis with ELK for Business Intelligence systems

Tue, 28 Feb 2017 21:43:41 +0000

In this post I’ll show howto collect logs from several applications (Oracle OBIEE, Oracle Essbase, QlikView, Apache logs, Linux system logs) with the ELK (Elasticsearch, Logstash and Kibana) stack. ELK is a powerful opensource alternative for Splunk. It can easily manage multiline logs. Installing the ELK stack in docker containers is really fast, easy and flexible..

Client side

To be installed where are the logs. On Linux, using docker, run

docker run -d --restart=unless-stopped -v "$PWD"/filebeat.yml:/filebeat.yml -v "$PWD"/filebeat.template.json:/filebeat.template.json -v "$PWD"/filebeat.template-es2x.json:/filebeat.template-es2x.json -v /home/oracle/Oracle:/Oracle:ro -v /var/log:/var/log:ro --name filebeat prima/filebeat:latest

Sample filebeat.yml file for collecting Obiee logs is

filebeat.prospectors:
- input_type: log
  document_type: obiee_access_log
  paths:
	- /Oracle/Middleware/Oracle_Home/user_projects/domains/bi/servers/bi_server1/logs/access.log
  exclude_lines: ["\t200\t[0-9]+$", "\t304\t[0-9]+$"]
- input_type: log
  document_type: obiee_log
  paths:
	- /Oracle/Middleware/Oracle_Home/user_projects/domains/bi/servers/bi_server1/logs/bi_server1-diagnostic.log
	- /Oracle/Middleware/Oracle_Home/user_projects/domains/bi/servers/obiccs1/logs/nqcluster.log
	- /Oracle/Middleware/Oracle_Home/user_projects/domains/bi/servers/obiccs1/logs/nqscheduler.log
	- /Oracle/Middleware/Oracle_Home/user_projects/domains/bi/servers/obijh1/logs/jh.log
	- /Oracle/Middleware/Oracle_Home/user_projects/domains/bi/servers/obips1/logs/sawlog0.log
	- /Oracle/Middleware/Oracle_Home/user_projects/domains/bi/servers/obis1/logs/obis1-diagnostic.log
	- /Oracle/Middleware/Oracle_Home/user_projects/domains/bi/servers/obisch1/logs/nqscheduler.log
  multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
  ignore_older: 10m
  fields:
	level: debug
- input_type: log
  paths:
	- /var/log/*.log
	- /var/log/messages
  document_type: syslog
  exclude_lines: ["^DBG"]
  include_lines: ["^ERR", "^WARN", "Error", "error"]
  exclude_files: [".gz$", "vmware-vmsvc*.log"]
  fields:
	level: debug
  ignore_older: 10m
name: zaff.redaelli.org
tags: ["obiee"]
fields:
  env: production
output.logstash:
  hosts : ["fourier.redaelli.org:5044"]
processors:
- drop_event:
	 when:
	   or:
		 - regexp:
			 message: "/Info"
		 - regexp:
			 message: "NOTIFICATION"
logging.level: error
logging.selectors: ["*"]

Sample filebeat.yaml for Qlikview logs

filebeat.prospectors:
- input\_type: log
 document\_type: qlikview\_log
 paths:
 - G:\\QLIK\_CONFIG\\QVS\\Logs\\Events\_APP13IT\_\*.log
 include\_lines: \["Error"\]
 ignore\_older: 10m
 fields:
 name: app13it
- input\_type: log
 document\_type: qlikview\_log
 paths:
 - G:\\QLIK\_CONFIG\\QVS\\Logs\\Events\_APP18IT\_\*.log
 include\_lines: \["Error"\]
 ignore\_older: 10m
 fields:
 name: app18it
- input\_type: log
 document\_type: qlikview\_log
 paths:
 - G:\\QLIK\_CONFIG\\QVS\\Logs\\Events\_APP25IT\_\*.log
 include\_lines: \["Error"\]
 ignore\_older: 10m
 fields:
 name: app25it
#================================ General =====================================
tags: \["qlikview"\]
fields:
 env: production

#================================ Outputs =====================================
output.logstash:
 # The Logstash hosts
 hosts : \["ambra.redaelli.org:5044"\]
#================================ Logging =====================================
logging.level: error

Server side (Logstash and kibana)

On linux, using docker

docker run -d --restart=unless-stopped --name elasticsearch -p 9200:9200 -p 9300:9300 -v /home/oracle/apps/docker\_shares/elasticsearch\_data:/usr/share/elasticsearch/data elasticsearch
docker run -d --restart=unless-stopped --name logstash -p 5044:5044 --link elasticsearch:elasticsearch -v /home/oracle/apps/docker\_shares/logstash:/logstash logstash -f /logstash/logstash.conf
docker run --restart=unless-stopped --name kibana --link elasticsearch:elasticsearch -p 5601:5601 -d kibana

Sample logstash.conf

input {
 beats {
 port => 5044
 }
}

filter {
  if [type] == "syslog" {
	grok {
	  match => { "message" => "%{SYSLOGLINE}" }
	}

	date {
	  match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
	}
  } else
  if [type] == "apache_httpd_log" {
	grok {
	  match => { "message" => "%{COMBINEDAPACHELOG}" }
	}
	date {
	  match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
	}
  } else
  if [type] == "essbase_log" {
	mutate {
	   gsub => ['message', "\n", " "]
	}
	mutate {
		gsub => ['message', "\r", " "]
	}
	grok{
		match => ["message","\[%{TIMESTAMP_ISO8601:timestamp}\] \[.*\] \[%{LOGLEVEL:loglevel}\:?\d*\] %{GREEDYDATA:log_message}"]
	}
	date {
		match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZZ"]
	}
	if ([message] !~ "ERROR" or [message] !~ "Error") {
	  drop {}
	}
  } else
  if [type] == "obiee_log" {
	mutate {
	   gsub => ['message', "\n", " "]
	}
	mutate {
		gsub => ['message', "\r", " "]
	}
	grok{
		match => ["message","\[%{TIMESTAMP_ISO8601:timestamp}\] \[.*\] \[%{LOGLEVEL:loglevel}\:?\d*\] %{GREEDYDATA:log_message}"]
	}
	date {
		match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZZ"]
	}
	if ([message] !~ "\[ERROR") {
	  drop {}
	}
  } else
  if [type] == "obiee_access_log" {
	 # 2017-02-17	11:37:01	GET	e82f1268-7238-4672-a983-03e62ebc9b9c-0000026f	0	/analytics/Missing_uicomponents/sensecomplete/sensecomplete.css	404	1164
	grok {
		match => { "message" => "%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}\t%{TIME:time}\t%{WORD:method}\t%{NOTSPACE:dummy1}\t%{NUMBER:dummy2}\t%{URIPATH:uri}\t%{NUMBER:response}\t%{NUMBER:bytes}"}
	}
	mutate {
	  add_field => {
		"timestamp" => "%{year}-%{month}-%{day} %{time}"
	  }
	}
	mutate {
		gsub => ['message', "\t", " "]
	}
	date {
	  match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
	  timezone => "Europe/Rome"
	}
	mutate {
	  remove_field => [ "timestamp", "year", "month", "day", "time", "dummy1", "dummy2" ]
	}
  } else
  if [type] == "qlikview_log" {
	# Server Started	Timestamp	SeverityID	EventID	Severity	Message
	# 2017-02-19 18:01:29	2017-02-21 00:00:16	4	700	Information	Debug: CNTService::Handler(4)
	grok {
		match => { "message" => "%{YEAR:y}-%{MONTHNUM:m}-%{MONTHDAY:d} %{TIME:t}\t%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day} %{TIME:time}\t%{NUMBER:severityid}\t%{NUMBER:eventid}\t%{WORD:loglevel}\t%{GREEDYDATA:log_message}"}
	}
	mutate {
	  add_field => {
		"timestamp" => "%{year}-%{month}-%{day} %{time}"
	  }
	}
	date {
	  match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
	  timezone => "Europe/Rome"
	}
  }
}

output {
   elasticsearch {
	  hosts => ["elasticsearch:9200"]
	  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
   }
   stdout {
	  codec => rubydebug
   }
}

Sample microservice for exposing database tables via REST using node.js express.js in a docker container

Thu, 09 Feb 2017 17:17:27 +0000

I have started learning node.js: The first result is this github repository where you can find a basic project of a rest web service that exposes oracle data. How to run the service forever (with autorestart on failures)? I tested it both with a docker container and the pm2 tool

Building a Chat Bot...

Sat, 21 Jan 2017 15:59:31 +0000

[last update: June, 5 2017] My next project will be a Chat Bot.. Starting points are:

AIML (Artificial Intelligence Markup Language) standard
https://messengerplatform.fb.com/ [from Facebook]
https://dev.botframework.com/ [from Microsoft]
https://www.rivescript.com/
https://sourceforge.net/projects/chatscript/
Chatterbot

There are also some useful softwares as a service like

Api.ai (Google)
Wit.ai (Facebook)
LUIS (Microsoft)
Watson (IBM)
Lex (Amazon)

And alternatives:

rasa.ai

Useful ML libraries:

Managing Spark dataframes in Python

Fri, 02 Dec 2016 11:00:41 +0000

Below a quick sample of using Apache Spark (2.0) dataframes for manipulating data. Sample data is a file of jsonlines like``` {“description”: “255/40 ZR17 94W”, “ean”: “EAN: 4981910401193”, “season”: “tires_season summer”, “price”: “203,98”, “model”: “Michelin Pilot Sport PS2 255/40 R17”, “id”: “MPN: 2351610”} {“description”: “225/55 R17 101V XL”, “ean”: “EAN: 5452000438744”, “season”: “tires_season summer”, “price”: “120,98”, “model”: “Pirelli P Zero 205/45 R17”, “id”: “MPN: 530155”}

from pyspark.sql import SparkSession
from pyspark.sql import Row
from pyspark.sql.functions import col
from pyspark.sql.functions import lit
from pyspark.sql.functions import \*
import re, sys


# warehouse\_location points to the default location for managed databases and tables
warehouse\_location = 'spark-warehouse'

spark = SparkSession \\
    .builder \\
    .appName("Python Spark  ") \\
    .config("spark.sql.warehouse.dir", warehouse\_location) \\
    .enableHiveSupport() \\
    .getOrCreate()

records\_orig = spark.read.json("scraped\_tyres\_data.json")

## removing bad records 
records = records\_orig \\
  .filter(records.id != '') \\
  .filter(regexp\_extract('description', '(rinnovati)', 1) == '')

## saving bad records  
records\_orig.subtract(records).coalesce(1).write.csv("bad-records.csv", sep=";")

# extract new features
regexp\_size = "(\\d+)/(\\d+) R(\\d+) (\\d+)(\\w+)\\s\*"

records = records \\
  .withColumn("width",       regexp\_extract("description", regexp\_size, 1)) \\
  .withColumn("ratio",       regexp\_extract("description", regexp\_size, 2)) \\
  .withColumn("diameter",    regexp\_extract("description", regexp\_size, 3)) \\
  .withColumn("load\_index",  regexp\_extract("description", regexp\_size, 4)) \\
  .withColumn("speed\_index", regexp\_extract("description", regexp\_size, 5)) \\
  .withColumn("brand",       regexp\_extract("model", "^(\\w+) ", 1)) \\
  .withColumn("season",      trim(regexp\_replace("season", "tires\_season",""))) \\
  .withColumn("id",          trim(regexp\_replace("id", "MPN: ",""))) \\
  .withColumn("ean",         trim(regexp\_replace("ean", "EAN: ",""))) \\
  .withColumn("runflat",     regexp\_extract("description", "(runflat)", 1)) \\
  .withColumn("mfs",         regexp\_extract("description", "(MFS|FSL|bordo di protezione|bordino di protezione)", 1)) \\
  .withColumn("xl",          regexp\_extract("description", " (XL|RF)\\s\*", 1)) \\
  .withColumn("chiodabile",  regexp\_extract("description", "(chiodabile)\\s\*", 1))

## extracting and saving all season values
records.select("season").distinct().coalesce(1).write.csv("season\_values", sep=";")

# misc
# records.columns   # show columns
# records.groupBy("brand").count().show()
# records.groupBy("brand").count().filter("count > 100").show(20,False)
#
# renaming all columns before joining dataframes with same column names
# records\_renamed = records.select(\*(col(x).alias(x + '\_renamed') for x in records.columns))
# join two dataframe
# records.join(record\_renamed, records.ean == records\_renamed.ean\_renamed)
#
#
# saving data to several formats
records.coalesce(1).write.csv("result.csv", sep=";")
records.write.json("result.json")
records.write.parquet("result.parquet")
records.write.format("com.databricks.spark.avro").save("result.avro")


```

Howto batch Install OBIEE 12c (silent mode)

Thu, 24 Nov 2016 12:24:00 +0000

If you wanted to install / deploy automatically obiee systems in a datacenter/cloud you could simply run few simple commands like:``` export TEMP=/home/oracle/tmp export TEMPDIR=/home/oracle/tmp export JAVA_HOME=/home/oracle/apps/jdk1.8.0

java -Djava.io.tmpdir=/home/oracle/tmp -jar fmw_12.2.1.2.0_infrastructure.jar \ -silent -responseFile /home/oracle/KIT/response_01_fmw_infrastructure.rsp \ -invPtrLoc /home/oracle/oraInst.loc

./bi_platform-12.2.1.2.0_linux64.bin -silent \ -responseFile /home/oracle/KIT/response_02_bi_platform.rsp \ -invPtrLoc /home/oracle/oraInst.loc \ -ignoreSysPrereqs

export ORACLE_HOME=/home/oracle/Oracle/Middleware/Oracle_Home export BI_PRODUCT_HOME=$ORACLE_HOME/bi $BI_PRODUCT_HOME/bin/config.sh -silent \ -responseFile /home/oracle/KIT/response_03_bi_platform_config.rsp \ -invPtrLoc /home/oracle/oraInst.loc \ -ignoreSysPrereqs

---
### Comments:
#### 
[Rahul]( "panchalrahul6@gmail.com") - Dec 4, 2018

from where to get those silent files ?

Any faster alternative to #Hadoop HDFS?

Thu, 17 Nov 2016 08:36:05 +0000

I’d like to have an alternative to Hadoop HDFS, a faster and not java filesystem:

S3: S3 Support in Apache Hadoop if your servers are hosted at Amazon AWS
chep: using hadoop with ceph
glusterfs: managing hadoop compatible storage
lustre: Running hadoop with lustre
Openstack Swift: Hadoop OpenStack Support: Swift Object Store
xstreamfs: there is an hadoop client

Which is better? Any suggestions? References:

[1] https://en.wikipedia.org/wiki/Comparison_of_distributed_file_systems

Apache Spark howto import data from a jdbc database using python

Thu, 27 Oct 2016 13:18:32 +0000

Using Apache spark 2.0 and python I’ll show how to import a table from a relational database (using its jdbc driver) into a python dataframe and save it in a parquet file. In this demo the database is an oracle 12.x file jdbc-to-parquet.py:``` from pyspark.sql import SparkSession

spark = SparkSession \ .builder \ .appName(“Python Spark SQL basic example”) \ .getOrCreate()

df = spark.read.format(“jdbc”).options(url=“jdbc:oracle:thin:ro/ro@mydboracle.redaelli.org:1521:MYSID”, dbtable=“myuser.dim_country”, driver=“oracle.jdbc.OracleDriver”).load()

df.write.parquet(“country.parquet”)

Mailman: Howto recover a mailinglist admin password

Mon, 09 May 2016 11:28:54 +0000

Do you use mailman and have you lost your admin password of you mailman list? You can easily reset it with``` sudo /usr/lib/mailman/bin/change_pw -l listname -p newpass

Building websites for high traffic with REST APIs, AngularJs and Jekyll

Thu, 05 May 2016 21:19:57 +0000

If you have few hw resources and/or you expect high traffic on your website, here are some quick suggestions (also taken from the article Meet the Obama campaign’s $250 million fundraising platform):

Expose your business logic with REST services
Use a javascript framework like angularJS for calling your rest APIs and building a dynamic site
Build a (not so) static website using Jekyll or similars and put your static files on S3 (if you are using Amazon AWS)
use a CDN

A sample website (not hosted at AWS but home based using a poor raspberryPI2) is http://paroleonline.it

Adding an application (angularjs+rest api) inside a Wordpress site

Tue, 26 Apr 2016 19:33:03 +0000

If you need to integrate an application written with AngularJS and Rest API services in your wordpress website, just create an empy page and edit it in “text” mode with something like```

DDNS for your GoDaddy domains

Wed, 13 Apr 2016 20:04:52 +0000

You can automatically update DNS for your GoDaddy.com domains following the suggestions from the post ddns-with-godaddy but remember to patch the godaddy python library with this client.py

Installing Nodejs oracledb module on Suse SLES 11

Thu, 11 Feb 2016 09:52:30 +0000

For a quick tutorial about installing Oracle module for Nodejs (oracledb) on Suse SLES, follow the info at Node-OracleDB Installation but remember to use the gcc compiler release 5.0``` export ORACLE_HOME=/home/oracle/instantclient_12_1 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME export TNS_ADMIN=$ORACLE_HOME export OCI_LIBRARY_PATH=$ORACLE_HOME export OCI_LIB_DIR=$ORACLE_HOME export OCI_INC_DIR=$ORACLE_HOME/sdk/include

CC=gcc-5 CXX=g++-5 npm install oracledb
```

Deploying microservices in a Docker container

Fri, 11 Dec 2015 13:09:29 +0000

I already spoke about docker containers (moving datacenters apps from virtual machines to containers) This is a quick tutorial (my github sample code) about a new way of deploying (micro) services and applications, ie using Docker containers: a sample python webservice and an simple web (html + angularJS code) page Creating docker containers means defining a file Dockerfile like``` FROM python:3.5 #FROM python:3-onbuild

ENV DEBIAN_FRONTEND noninteractive

ENV HTTP_PROXY=“http://myproxy.redaelli.org:80” ENV HTTPS_PROXY=“http://myproxy.redaelli.org:80” ENV http_proxy=“http://myproxy.redaelli.org:80” ENV https_proxy=“http://myproxy.redaelli.org:80” ENV PIP_OPTIONS="–proxy $HTTP_PROXY"

COPY requirements.txt /usr/src/app/ COPY app.py /usr/src/app/

WORKDIR /usr/src/app RUN apt-get update && apt-get install -y nmap RUN pip install –proxy $HTTP_PROXY –no-cache-dir -r requirements.txt

VOLUME ["/usr/src/app"] EXPOSE 5000

ENTRYPOINT [“python”] CMD ["./app.py"]

Put the additional python packages you need in a file requirements.txt Flask python-nmap dnspython3 And create your application in the file app.py In this way we are going to create a docker container with python3 and some additional python packages with the command docker build -t python-infra-ws . Finally we'll start the container with the command docker run -d -t –name python-infra-ws -p 5000:5000 python-infra-ws Some other useful commands are: docker stop python-infra-ws docker start python-infra-ws docker ps python-infra-ws docker rm python-infra-ws docker rmi python-infra-ws

Download 1 million of twitter users

Mon, 07 Dec 2015 17:06:54 +0000

I like learning and using new technologies and opensource softwares. For instance I use Apache Camel and MongoDB for downloading and analyzing twitter data. Here you can download a sample file with (some attributes of) 1 million of twitter users

Analyzing huge sensor data in near realtime with Apache Spark Streaming

Wed, 25 Nov 2015 21:19:58 +0000

For this demo I downloaded and installed Apache Spark 1.5.1 Suppose you have a stream of data from several (industrial) machines likeMACHINE,TIMESTAMP,SIGNAL1,SIGNAL2,SIGNAL3,... 1,2015-01-01 11:00:01,1.0,1.1,1.2,1.3,.. 2,2015-01-01 11:00:01,2.2,2.1,2.6,2.8,. 3,2015-01-01 11:00:01,1.1,1.2,1.3,1.3,. 1,2015-01-01 11:00:02,1.0,1.1,1.2,1.4,. 1,2015-01-01 11:00:02,1.3,1.2,3.2,3.3,.. ...Below a system, written in Python, that reads data from a stream (use the command “nc -lk 9999” to send data to the stream) and every 10 seconds collects alerts from signals: at least 4 suspicious values of a specific signal of the same machine``` from pyspark import SparkContext from pyspark.streaming import StreamingContext

min_occurs = 4

def signals_from_1_row_to_many(row): “output is (machine, date, signal_number, signal_value)” result = [] for f in range(2,21): result = result + [(row[0], row[1], f-1, row[f])] return result

def isAlert(signal, value): defaults = [83.0, 57.0, 37.0, 57.0, 45.0, 19.0, -223.0, 20.50, 20.42, 20.48, 20.24, 20.22, 20.43, 20, 20.44, 20.39, 20.36, 20.25, 1675.0] soglia = 0.95 if value == ‘’: return True value = float(value) ref = defaults[signal -1] if value < ref - soglia*ref or value > ref + soglia*ref: return True else: return False def isException(machine, signal):

sample data. the sensor 19 of machine 11 is broken

exceptions = [(11,19)] return (int(machine), signal) in exceptions

Create a local StreamingContext with two working thread and batch interval of 10 second

sc = SparkContext(“local[2]”, “SignalsAlerts”) ssc = StreamingContext(sc, 10)

Create a DStream that will connect to hostname:port, like localhost:9999

lines = ssc.socketTextStream(“localhost”, 9999)

all_alerts = lines.map(lambda l: l.split(",")) \ .flatMap(signals_from_1_row_to_many) \ .filter(lambda s: isAlert(s[2], s[3])) \ .filter(lambda s: not isException(s[0], s[2])) \ .map(lambda s: (s[0]+’-’+str(s[2]), [(s[1], s[3])])) \ .reduceByKey(lambda x, y: x + y)

alerts = all_alerts.filter(lambda s: len(s[1]) > min_occurs)

alerts.pprint()

ssc.start() # Start the computation ssc.awaitTermination() # Wait for the computation to terminate

TwitterPopularTags.scala example of Apache Spark Streaming in a standalone project

Thu, 22 Oct 2015 20:12:01 +0000

This is an easy tutorial of using Apache Spark Streaming with Scala language using the official TwitterPopularTags.scala example and putting it in a standalone sbt project. In few minutes you will be able to receive streams of tweets and manipulating then in realtime with Apache Spark Streaming

Install Apache Spark (I used 1.5.1)
Install sbt
git clone https://github.com/matteoredaelli/TwitterPopularTags
cd TwitterPopularTags
cp twitter4j.properties.sample twitter4j.properties
edit twitter4j.properties
sbt package
spark-submit –master local –packages “org.apache.spark:spark-streaming-twitter_2.10:1.5.1” ./target/scala-2.10/twitterpopulartags_2.10-1.0.jar italy

Howto collecting twitter data in 15 minutes

Fri, 16 Oct 2015 14:01:05 +0000

For this tutorial I assume you are using a Debian/Ubuntu Linux system but it could be easily adapted for other Openrating Systems

Install the software

apt-get install openjdk-7-jdk  
wget [http://apache.panu.it/karaf/4.0.2/apache-karaf-4.0.2.tar.gz](http://apache.panu.it/karaf/4.0.2/apache-karaf-4.0.2.tar.gz)
tar xvfz apache-karaf-4.0.2.tar.gz

Start the server

cd apache-karaf-4.0.2/
./bin/start

Install additional connectors

ssh -p 8101 karaf@localhost
feature:repo-add camel 2.16.0
feature:install camel camel-blueprint camel-twitter camel-jackson camel-dropbox
exit

Configure our routes

Create two new files: twitter-to-file.xml```

http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi=“http://www.w3.org/2001/XMLSchema-instance" xmlns:camel=“http://camel.apache.org/schema/blueprint" xsi:schemaLocation=” http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd http://camel.apache.org/schema/blueprint http://camel.apache.org/schema/blueprint/camel-blueprint.xsd">

${in.header.twitter-id} ${body}

${body}\n ${in.header.CamelFileName}/${date:now:yyyy}/${date:now:MM}/${date:now:dd} twitter-streaming-sample.xml sample

About Cayley a scalable graph database

Mon, 03 Aug 2015 13:17:24 +0000

This is fast tutorial of using the Caylay graph database (with MongoDB as backend): Cayley is “not a Google project, but created and maintained by a Googler, with permission from and assignment to Google, under the Apache License, version 2.0”

download and unzip a binary distribution
edit cayley.cfg

{
"database": "mongo",
"db\_path": "cayley.redaelli.org:27017",
"read\_only": false,
"host": "0.0.0.0"
}

./cayley init -config=cayley.cfg
./cayley http -config=cayley.cfg -host=“0.0.0.0” &
create a file demo.n3

"/user/matteo" "is\_manager\_of" "/user/ele" .
"/user/matteo" "has" "/workstation/wk0002" .
"/user/matteo" "lives\_in" "/country/italy" .

upload data with: curl http://cayley.redaelli.org:64210/api/v1/write/file/nquad -F NQuadFile=@demo.n3
or: ./cayley load –config=cayley.cfg -quads=demo.n3
query data with: curl –data ‘g.V("/user/matteo").Out(null,“predicate”).All()’ http://cayley.redaelli.org:64210/api/v1/query/gremlin

{
 "result": \[
  {
   "id": "/workstation/wk0002",
   "predicate": "has"
  },
  {
   "id": "/country/italy",
   "predicate": "lives\_in"
  },
  {
   "id": "/user/ele",
   "predicate": "is\_manager\_of"
  }
 \]

A comparison of Hadoop distributions

Thu, 21 May 2015 14:39:17 +0000

Read all the article at http://www.hadoop360.com/blog/hadoop-whose-to-choose

Howto Upgrade, Migrate & Consolidate to Oracle Database 12c

Tue, 12 May 2015 15:00:37 +0000

I suggest to visit the site https://blogs.oracle.com/UPGRADE and the attachment pdf Upgrade, Migrate & Consolidate to Oracle Database 12c

Powershell: how to retreive service accounts from services for many servers

Mon, 11 May 2015 15:28:02 +0000

Below a sample script``` $hosts = Get-Content “hosts.txt” foreach ($myhost in $hosts) { Get-WmiObject win32_service -ComputerName $myhost|where {$_.StartMode -eq “Auto”} | where {$_.StartName -like “*.*”} | foreach {write-host $myhost, $_.Startname} }

Apache Spark news from a Spark Summit 2015

Tue, 21 Apr 2015 20:12:00 +0000

GOAL: unified engine across data sources, workloads and environments.

Highlights: dataframes (1.3), SparkR (1.4), …

See all video and slides at http://spark-summit.org

Hortonworks, IBM and Pivotal begin shipping standardized Hadoop

Wed, 15 Apr 2015 09:49:30 +0000

“Hortonworks, IBM and Pivotal begin shipping standardized Hadoop. The standardization effort is part of the Open Data Platform initiative, which is an industry effort to ensure all versions of Hadoop are based on the same Apache core..”. Read all the full zdnet.com article This is thanks to the Apache Ambari project!

Cloudera still goes alone with its old custom solution.
And MapR bets on Apache Mesos (see https://lnkd.in/e_ZwNn5)

I also suggest to install Ambari/Hadoop in a Docker container.. the new wave of “virtualization”. About Docker & Hadoop: Hortonworks to Acquire SequenceIQ to Speed Hadoop Deployments into the Cloud (docker containers) http://investors.hortonworks.com/phoenix.zhtml?c=253804&p=irol-newsArticle&ID=2034442

Howto export Oracle Essbase databases with MaxL / essmsh commands

Fri, 10 Apr 2015 08:27:26 +0000

Manually``` essbase@olap-server:~> /opt/essbase/Oracle/Middleware/EPMSystem11R1/products/Essbase/EssbaseServer/templates/startMaxl.sh

MAXL> login Hypadmin mypassword on olap-server.redaelli.org;

OK/INFO - 1051034 - Logging in user [Hypadmin@Native Directory]. OK/INFO - 1241001 - Logged in to Essbase.

MAXL> export database P_BSO.Plan1 level0 data to data_file ‘ExpLev0_P_BSO.Plan1’;

OK/INFO - 1054014 - Database Plan1 loaded. OK/INFO - 1051061 - Application P_BSO loaded - connection established. OK/INFO - 1054027 - Application [P_BSO] started with process id [60396]. OK/INFO - 1019020 - Writing Free Space Information For Database [Plan1]. OK/INFO - 1005031 - Parallel export completed for this export thread. Blocks Exported: [2013908]. Elapsed time: [312.35].. OK/INFO - 1005002 - Ascii Backup Completed. Total blocks: [2.01391e+06]. Elapsed time: [312.35].. OK/INFO - 1013270 - Database export completed [‘P_BSO’.‘Plan1’].

Automatically /opt/essbase/Oracle/Middleware/EPMSystem11R1/products/Essbase/EssbaseServer/templates/startMaxl.sh -u Hypadmin -p mypassword -s localhost backup-databases.msh with a file backup-databases.msh like export database P_BSO.Plan1 level0 data to data_file ‘ExpLev0_P_BSO.Plan1’; BUT if you need to **export both metadata and data**, you should run the command MAXL> alter database P_BSO_D.Plan1 force archive to file ‘P_BSO_D.Plan1.arc’;

Archlinux and Docker for my Raspberry PI2

Sat, 28 Mar 2015 21:30:49 +0000

What is the best linux distribution for Raspberry PI2? I started with Raspian (Debian is my preferred Linux distribution for servers, desktops and laptops) but docker didn’t work. But with Archlinux it works fine. How to create a docker images with Archlinux & RPI2? See http://linucc.github.io/docker-arch-rpi2/ matteoredaelli/docker-karaf-rpi is the first docker image I have created. Below my docker info output:

[root@raspi1 ~]# docker info Containers: 4 Images: 9 Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 17 Execution Driver: native-0.2 Kernel Version: 3.18.10-1-ARCH Operating System: Arch Linux ARM CPUs: 1 Total Memory: 432.8 MiB Name: raspi1 ID: UOFM:E7CP:2OTL:VTTM:QRP2:JNJ7:UFCI:2MDE:AOYQ:MGTM:Q25F:FL37 WARNING: No memory limit support WARNING: No swap limit support

Below some docker survival commands:``` docker run -i -t –name karaf \ -p 1099:1099 -p 8101:8101 \ -p 44444:44444 -v /apps/karaf-deploy:/deploy \ matteoredaelli/karaf-docker-rpi /bin/bash docker start karaf docker stop karaf docker exec -it karaf bash docker top docker ps docker ps -a docker images

A case study of adopting Bigdata technologies in your company

Fri, 13 Mar 2015 20:52:33 +0000

Bigdata projects can be very expensive and can easily fail: I suggest to start with a small, useful but not critical project. Better if it is about unstructured data collection and batch processing. In this case you have time to get practise with the new technologies and the Apache Hadoop system can have not critical downtimes. At home I have the following system running on a small Raspberry PI: for sure it is not fast ;-) At work I introduced Hadoop just few months ago for collecting web data and generating daily reports.

QlikView: how to share bookmarks

Tue, 10 Mar 2015 16:07:28 +0000

In your QlikView report, select the “Repository” button (the first from the right) and then select “Bookmarks”…

Before SQL then NOSQL and BIGDATA: now BIGDATA and SQL again

Mon, 02 Mar 2015 13:37:35 +0000

The trend of these years has been switching from SQL (RDBMS) databases to NoSQL databases like Hadoop, MongoDB, Cassandra, Riak, … SQL is a old but easy and fast way to query data. And people STILL look at it for quering Hadoop and bigdata:

Read details from 10 ways to query hadoop with sql ..

Bigdata on Twitter 2015-02

Mon, 02 Mar 2015 11:00:34 +0000

Below the top media (and other statistics) about “bigdata” word in Twitter in February 2015

Expo2015 on Twitter 2015-02

Mon, 02 Mar 2015 10:52:53 +0000

Below the top media (and other statistics) about “Expo2015” search in Twitter in February 2015

Hadoop on Twitter 2015-02

Mon, 02 Mar 2015 10:03:30 +0000

Below the top media (and other statistics) about “Hadoop” search in Twitter in February 2015

Statistics of #Hadoop #Opensource #Expo2015 tweets in January 2015

Thu, 05 Feb 2015 13:05:47 +0000

How easy is analyzing json twitter data using Apache Spark and Apache Hadoop. Below some examples Tweets about Expo2015 Tweets about Hadoop Tweets about opensource

Howto managing tweets saved in #Hadoop using #Apache #Spark SQL

Thu, 15 Jan 2015 16:35:40 +0000

Instead of using the old Hadoop way (map/reduce), I suggest using the newer and faster way (Apache Spark on top of Hadoop Yarn): in few lines you can open all tweets (zipped json files saved in several subdirectories hdfs://path/to/YEAR/MONTH/DAY/*gz) and query them in a SQL like language``` sc = SparkContext(appName=“extraxtStatsFromTweets.py”) sqlContext = SQLContext(sc) tweets = sqlContext.jsonFile("/tmp/twitter/opensource/2014/*/*.gz") tweets.registerTempTable(“tweets”) t = sqlContext.sql(“SELECT distinct createdAt,user.screenName,hashtagEntities FROM tweets”) tweets_by_days = count_items(t.map(lambda t: javaTimestampToString(t[0]))) stats_hashtags = count_items(t.flatMap(lambda t: t[2])\ .map(lambda t: t[2].lower()))

Howto managing tweets saved in #Hadoop using #Apache #Spark

Tue, 25 Nov 2014 20:41:10 +0000

Apache Spark has just passed Hadoop in popolarity on the web (google trends) My first Apache Spark usage was extracting texts from tweets I’ve been collecting in Hadoop HDFS. My python script tweet-texts.py was``` import json

from pyspark import SparkContext

def valid(tweet): return ’text’ in tweet

def gettext(line): tweet = json.loads(line) return tweet[’text’]

sc = SparkContext(appName=“Tweets”) data = sc.textFile(“hdfs://hadoop.redaelli.org:9000/user/matteo/staging/twitter/searches/TheCalExperience.json/*/*/*.gz”)

result = data.filter(lambda line: valid(line))\ .map(lambda tweet: gettext(tweet))

output = result.collect() for text in output: print text.encode(‘utf-8’) And lunched with spark-1.1.0> bin/spark-submit –master local[4] tweet-texts.py I used Apche Spark 1.1.0 and Apache Hadoop 2.5.2. When I compiled Spark with mvn -Phadoop-2.5 -Dhadoop.version=2.5.2 -DskipTests -Pyarn -Phive package I got an error related to protocolBuffer jar release when I tried to read files from Hadoop HDFS org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$CreateSnapshotRequestProto overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet So I changed the pom.xml adding hadoop-2.5 2.5.2 2.5.0 0.9.0

Workflows in Apache Hadoop

Fri, 03 Oct 2014 20:20:43 +0000

How to orchestrate your Hadoop Jobs? Possible solutions are:

Apache Oziee included in the top Hadoop distributions
Azkaban from LinkedIn
Luigi from Spotify
Apache Airflow from AirBnb

See for instance a comparison among luigi, airflow and pinball at http://bytepawn.com/luigi-airflow-pinball.html

#Hadoop Search with Apache #Solr

Fri, 03 Oct 2014 19:35:08 +0000

The two top Hadoop distributions (Cloudera and Hortonworks but remember that Hadoop is a Free Software and many companies do not pay anything for using it!) include Apache Solr as Hadoop search tool See apache-solr-hadoop-search article and the following two presentations from the two vendors [slideshare id=35810888&doc=solr-2-140612162029-phpapp02] [slideshare id=24255985&doc=hadoopplussolrbigdatasearch-130715115557-phpapp02] See also the Natural Language Processing and Sentiment Analysis for Retailers using HDP and ITC Infotech Radar article

Adding Google Analytics to Oracle OBIEE

Mon, 29 Sep 2014 11:39:23 +0000

I opened a service request to Oracle and they did not provide me an official way to add the Google Analytics javascript code to Oracle OBIEE (release 11.1.1.7): I wanted to add it in only one place and see it in all pages of Oracle Obiee. The solution I found and tested is to add the javascript code (without and ) in the file``` bi_server1/tmp/_WL_user/analytics_11.1.1/7dezjl/war/res/b_mozilla/common.js

Google uses R: a new open-source package for estimating causal effects in time series

Tue, 16 Sep 2014 20:25:36 +0000

In this blog post Google confirms its adoption of the opensource statistical environment R (see my R introduction) releasing a new R package.. “How can we measure the number of additional clicks or sales that an AdWords campaign generated? How can we estimate the impact of a new feature on app downloads? How do we compare the effectiveness of publicity across countries? In principle, all of these questions can be answered through causal inference […] How the package works The CausalImpact R package implements a Bayesian approach to estimating the causal effect of a designed intervention on a time series. Given a response time series (e.g., clicks) and a set of control time series (e.g., clicks in non-affected markets, clicks on other sites, or Google Trends data), the package constructs a Bayesian structural time-series model with a built-in spike-and-slab prior for automatic variable selection. This model is then used to predict the counterfactual, i.e., how the response metric would have evolved after the intervention if the intervention had not occurred.” Read the full Google blog post

Machine learning map

Fri, 12 Sep 2014 19:28:45 +0000

I found this nice map about Machine Leaning and I wanted to share it. It is taken from http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html

Scouting howto collecting and analyzing sensor data with Hadoop or other NOSQL databases

Wed, 10 Sep 2014 19:44:01 +0000

Collecting and analyzing sensor data with hadoop or other no sql databases from Matteo Redaelli

Building a sample (micro) service in Prolog

Sun, 07 Sep 2014 19:41:26 +0000

Sometimes ago I read that some components of IBM Watson were implemented in prolog . So I decided to look at it again after many years… I like Prolog, I studied prolog at Computer Science University of Milan and for my thesis I wrote code in Prolog (and Lisp). proloGraph is a simple example of howto exposing a prolog graph database to other applications, building a REST web service. I used swi-prolog and its http library Install the prolog language (I used the fantastic Linux Debian distribution) withapt-get install swi-prologclone my git repositorygit clone https://github.com/matteoredaelli/proloGraph cd proloGraphRun it withswipl -s webserver.pl -g 'server(8765).'Open the following url with your browserhttp://localhost:8765/vertex?name=user(matteo)and you will get:``` { “prev”: [ {“from”:“user(gabriele)”, “to”:“user(matteo)”, “rel”:“follow”} ], “next”: [ {“from”:“user(matteo)”, “to”:“user(ele)”, “rel”:“follow”}, {“from”:“user(matteo)”, “to”:“user(gabriele)”, “rel”:“follow”}, {“from”:“user(matteo)”, “to”:“user(4)”, “rel”:“follow”}, {“from”:“user(matteo)”, “to”:“country(italy)”, “rel”:“lives”}, {“from”:“user(matteo)”, “to”:“hobby(running)”, “rel”:“likes”} ] }

Apache Pig for batch data analysis over Hadoop

Mon, 25 Aug 2014 20:13:24 +0000

In these days I’m playing with Apache Pig for running data analysis over Apache Hadoop. Below a sample wordcloud generated from the top word count of nouns of the Italian translation of the Bible Copy the file book.txt to hadoop distribuited file system (HDFS) withhadoop-2.4.0/bin/hdfs dfs -copyFromLocal -f book.txtTest the pig job locally withpig-0.13.0/bin/pig -x local wordcount.pigRun the pig job in hadoop withpig-0.13.0/bin/pig -x mapreduce wordcount.pigLook at results withhadoop-2.4.0/bin/hdfs dfs -cat book-wordcount/part\*|moreCopy the results to a local file withhadoop-2.4.0/bin/hdfs dfs -cat book-wordcount/part\* > frequenza-parole-bibbia.txt Below the two scripts I used for this short tutorial: Wordcount (pig script):a = load '/user/matteo/book.txt'; b = foreach a { line = LOWER(REPLACE((chararray)$0, '\[!?\\\\.»«:;,\\'\]', ' ')); generate flatten(TOKENIZE(line)) as word; } c = group b by word; d = foreach c generate group, COUNT(b) as cnt; d\_ordered = ORDER d BY cnt DESC; store d\_ordered into '/user/matteo/book-wordcount'; Wordcloud (R script)``` library(wordcloud) p = read.table(file=“frequenza-parole-bibbia.txt”) png("/home/matteo/la-sacra-bibbia-frequenza-parole.png", width=900, height=900) wordcloud(p$V1, p$V2, scale=c(8,.3),min.freq=2,max.words=200, random.order=T, rot.per=.15) dev.off()

Developing applications with a microservice architecture

Sat, 09 Aug 2014 19:14:39 +0000

“The microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery.” Read the full article This is my first sample microservices written in clojure

Free Online Timeseries forecasting tool

Sat, 09 Aug 2014 11:59:44 +0000

An Online demo for the Opensource tool (predictoR based on ltp R library) about forecasting / long term prediction of time series (any periods: daily, weekly, monthly, semesters, yearly,..) is available at rapid.tips!

Docker: moving datacenters/apps from Virtual Machines to containers

Mon, 04 Aug 2014 19:02:23 +0000

“Docker, a new container technology, is hotter than hot because it makes it possible to get far more apps running on the same old servers and it also makes it very easy to package and ship programs…” Read the full article at zdnet.com

#Cascading will support Apache #Tez, #Apache #Spark and Apache #Storm

Mon, 19 May 2014 19:51:27 +0000

In these nights I’m playing with Cascalog. for running (map reduce) jobs over Hadoop. A nice news for the future is that “Cascading 3.0 will initially ship with support for: local in-memory, Apache MapReduce (support for both Hadoop 1 and 2 are provided), and Apache Tez. Soon thereafter, with community support, Apache Spark™, Apache Storm and others will be supported through its new pluggable and customizable planner[…]”. Read the full InfoQ article

Top twitter users statistics

Tue, 13 May 2014 19:36:09 +0000

Starting collecting statistics about top (suggested) twitter users at site http://top-twitter-users.blogspot.it/

Statistiche politici italiani su twitter - aprile 2014

Tue, 13 May 2014 19:33:01 +0000

Dettagli al sito http://politici-italiani-sul-web.blogspot.it/

Politici italiani su twitter - marzo 2014

Wed, 02 Apr 2014 21:09:34 +0000

Playing with R, I extracted some data from twitter regarding Italian politicians. See the report of Politici Italiani su twitter marzo 2014

Apache Tez for Hadoop >= 2.0

Wed, 12 Mar 2014 16:19:16 +0000

The Apache Tez project is aimed at building an application framework which allows for a complex directed-acyclic-graph of tasks for processing data. It is currently built atop Apache Hadoop YARN. The 2 main design themes for Tez are:

Empowering end users by:

Expressive dataflow definition APIs

Flexible Input-Processor-Output runtime model

Data type agnostic

Simplifying deployment

Execution Performance
- Performance gains over Map Reduce
- Optimal resource management
- Plan reconfiguration at runtime
- Dynamic physical data flow decisions

Old way was: References:

#Hadoop : lift = #Storm : escalator

Sun, 16 Feb 2014 20:14:14 +0000

Apache Hadoop is not good for realtime computing: please evaluate Apache Storm (from twitter) or Apache Spark

References:

http://www.slideshare.net/edvorkin/storm-intro-v8

http://www.infoq.com/news/2014/01/Spark-Storm-Real-Time-Analytics

#Hadoop 1.X : @Cloudera = #Hadoop 2.X : @Hortonworks

Sat, 15 Feb 2014 17:19:52 +0000

Hortonworks.com

cloudera.com

dump_tweets.R a tool for saving tweets and user data to a database

Mon, 20 Jan 2014 19:57:38 +0000

dump_tweets.R is a tool for searching tweets and (recursively) crawl users from twitter: Data are then saved to a MySQL database and can finally be exported to .RData files dump_tweets.R is sponsorized by Associazione Rospo Example:

Rscript search.R -q “#opensource”

2014-01-20 20:53:43 INFO::Connecting to TWITTER… 2014-01-20 20:53:43 INFO::Connecting to database=twitter, host=localhost with user=root 2014-01-20 20:53:43 INFO::using UTF8 code 2014-01-20 20:53:43 INFO::Searching for q=#opensource, sinceID=0 2014-01-20 20:53:57 INFO::Found 191 tweets 2014-01-20 20:53:57 INFO::maxID=425355265857187841 2014-01-20 20:53:57 INFO::Saving data to tweet table… 2014-01-20 20:53:58 INFO::saving data to search_results table…

Retreiving and analyzing Facebook data with R

Thu, 21 Nov 2013 20:17:46 +0000

As suggested by Julianhi’s Blog post, I installed the CRAN Rfacebook package for the R statistical environment and I created a sample igraph with 20 of my “facebook friends”

## See examples for fbOAuth to know how token was created.

## Getting my network of friends

load(“fb_oauth”)

mat <- getNetwork(token=fb_oauth, format=“adj.matrix”)

library(igraph)

network <- graph.adjacency(mat, mode=“undirected”)

pdf(“network_plot.pdf”)

plot(network)

dev.off()

Comments:

Jbovnd - Dec 0, 2019

I do agree with all of the ideas you’ve offered to your post. They are really convincing and will definitely work. Still, the posts are very short for novices. May just you please prolong them a little from next time? Thank you for the post.

Hi there We are so thrilled I discovered yopur website, I definitely found you by error, while I was searching Yahoo for somethhing otherwise, Anyhow I amm below now and would simply like to ssay many thanks for a tremendous article and an all circular exciting blog (I furthdrmore love the theme/design), I actually don’t have time to be able to proceed through it all from the minute but My partner and i have saved it as well as added in yoiur RSS OR ATOM feeds, then when I pssess time I will get returning to read more, Remember to do carry on thhe amazikng job.

State of Linux Development 2013

Thu, 21 Nov 2013 15:56:10 +0000

From twitter @linuxfondation account

10,000 developers from 1,000 companies have contributed to #Linux since 2005.

Are you one of them? pic.twitter.com/6BIUgKDhnM

Howto unlock Alice Gate VoIP router

Wed, 25 Sep 2013 20:43:28 +0000

Following the suggestions from http://www.p2warticles.com/2012/04/menu-avanzato-telnet-alice-wifi-voip-2-plus/ I unlocked my Pirelli Broadband Solutions / Telecom Italia Alice Gate VoIP router agpf_4.5.2 with the following steps 1) downloading the backdoor tool (for linux and windows) 2) generating the payload using the mach address of my router

matteo@debian:~/backdoor_agpf_4.5.2$ ./backdoor 18:12:2S:E1:A1:B1

sending the payload to my router with the linux command (p=XXX where XXX is the payload previously generated)

mz -v eth0 -d 0 -t ip “proto=255,p=3a:b8:69:57:b6:6c:2a:a1”

After these few steps I was able to login via telnet to my router with the account admin/riattizzati and going on…

Google guides of R

Wed, 31 Jul 2013 14:07:16 +0000

Here you can find google lessons about R statistical language [youtube]http://www.youtube.com/watch?v=YhQOV27pQfg&list=PLOU2XLYxmsIK9qQfztXeybpHvru-TrqAP[/youtube]

How to Use Hadoop with your SAP Software Landscape

Wed, 31 Jul 2013 14:04:08 +0000

CIO Guide on Big Data “How to Use Hadoop with your SAP Software Landscape” just released

FirefoxOS on my Sony Xperia Live with Walkman (LWW)

Tue, 30 Jul 2013 19:13:17 +0000

[update 2013-11-06] how to install FireFoxOS 1.3

Unlock your phone,

download “flashtool” tool

download latest FireFoxOS build from here

flash boot.img included in the zip

copy the zip to the sdcard

boot in recovery mode and flash the zip

good lucks… ;-)

-————————-

First look at FirefoxOS on my Sony XPERIA 2011 Live With Walkman (LWW, wt19i, coconut) following suggestions from

http://www.galaxy4gaming.in/2013/07/firefox-os-for-xperia-2011-mini-and.html

Dump_tweets.R: how to collect interesting tweets

Sat, 06 Jul 2013 08:45:38 +0000

Do you need a simple R script for incremental downloading all tweets related an hashtag (or any words) and saving them to a database? If yes, look at dump_tweets.R project: feel free to clone my repository and push any fixes or new features Usage:

./dump_tweets.R –db /var/www/dump_tweets.R –add mysearches.csv

./dump_tweets.R –db /var/www/dump_tweets.R –show

./dump_tweets.R –db /var/www/dump_tweets.R

./dump_tweets.R –db /var/www/dump_tweets.R –remove opensource

Enjoy

How #ictspring 2013 looks like from Internet

Wed, 19 Jun 2013 20:56:06 +0000

Looking at tweets of day 1 (2013-06-19) of #ictspring 2013 I see

TOP CONTRIBUTORS

MOST USED DEVICES

GIVEN TOPICS

TWEETS DURING THE DAY

R for text mining & twitter

Wed, 12 Jun 2013 20:35:49 +0000

This is a sample report of topics of tweets related to

#ddaypirelli as suggested by miningtwitter website using the R statistical environment and some of its thousands libraries

Howto Install Android 4.4.2 to my Sony Live with walkman wt19i

Mon, 10 Jun 2013 20:04:36 +0000

UPDATES:

2014-03-25: look here at androidlegend.com for newer CarbonRom 4.4.2 builds

2014-01-04: see http://legacyxperia.github.io for unofficial builds of CyanogenMod 11.0 (Android 4.4.2)

2013-12-11: see http://legacyxperia.github.io for unofficial builds of CyanogenMod 11.0 (Android 4.4.1)

2013-09-01: see http://legacyxperia.github.io for unofficial builds of CyanogenMod 10.2 (Android 4.3) for legacy Sony Ericsson msm7x30 devices. It is an effort to join all developer forces to bring a stable cm-10.1 rom on these devices which are dropped from official CM

-.-.-

ORIGINAL_POST

After installing android tools

adb and fastboot

on my Linux desktop,

I installed Android 4.2.2 (Cyanogenmod distribution) following the steps suggested by the article

install-sony-live-with-walkman-wt19i-android-4-2

Varnish for speeding up your websites

Sat, 08 Jun 2013 19:40:20 +0000

“Varnish is able to serve objects out of its cache much faster and using far fewer host resources than a web server application..”

Read all the interesting article Adventures in Varnish

How to setup a multi node Hadoop cluster

Thu, 07 Mar 2013 20:46:58 +0000

Read the interesting articles by Michael G. Noll

Running Hadoop on Ubuntu Linux (Single-Node Cluster)

Running Hadoop on Ubuntu Linux (Multi-Node Cluster)

Enterprise search in the cloud (Solr & Amazon VPC)

Wed, 06 Mar 2013 16:33:32 +0000

“Recently [Harish Ganesan] architected a 13+ EC2 Node(s) SolrCloud Shard built in with HA and scalability inside Amazon Virtual Private Cloud. Since the base product engineered had both on-premise and SAAS edition, we were unfortunately not able to use Amazon CloudSearch as it may cause AWS dependency in this case in product tech stack. This Solr architecture was secured using Amazon VPC, Private subnets, Access controls, AWS Security groups and IAM. The SolrCloud cluster was designed to accommodate index data in memory and some search data overflowing to EBS disk. The Cluster was designed to handle ~100 GB of distributed index data and another 200+ GB of Search data on EBS disks. The Web/App tier, Business and Back ground tier will access the SolrCloud deployed in private subnet. In this post I will be sharing my experience, learning, best practice and tips on architecting SolrCloud on AWS platform.”

Read the full articles at

Introduction to Apache SolrCloud on AWS

Apache SolrCloud Implementation on Amazon VPC

Apache Camel: how to collect data from twitter

Sat, 29 Dec 2012 22:13:34 +0000

In the article Apache Camel: how to collect data from twitter,

I’ll show howto save all tweets about Pirelli to files in .json format ..

R statistical software in the Enterprise

Thu, 18 Oct 2012 20:57:29 +0000

I have updated my R presentation with the latest news (about Oracle, SAP, Tibco, teradata, IBM, ..) … Look at the presentation An opensource environment and language for statistics

Strategico 3.0 released!

Tue, 25 Sep 2012 21:00:32 +0000

Strategico 3.0 has been released!

Strategico is an opensource tool for making **forecasts and Long Term Predictions ** over a (huge) set of time series

Strategico is written with R, the most famous and used Statistical programming language

Howto quickly setup an interface among systems using Apache Camel / Karaf (OSGI)

Wed, 25 Jul 2012 18:04:56 +0000

In the article building system integrations with Apache Camel I’ll show how to create in 10 minutes an integration between two databases (without writing any lines of java or c# code):

looking for uses in the database MOODLE (mysql) with missing attributes
for each of that users retreiving the missing attributes from the database UPMS (m$ sql server) and then
adding the missing attributes to the database MOODLE

I’ll use

Apache Karaf 2.2.8
Apache Camel 2.10.0

under Linux. Any suggestions and comments are welcome! Matteo

London Olympics prediction for the 100m final with R and Strategico

Mon, 23 Jul 2012 08:03:05 +0000

After reading the interesting article London Olympics and a prediction for the 100m final here you can find an alternative statistical prediction using R and Strategico. Look at this excel for details

Using curl for querying Active Directory / LDAP from command line

Fri, 18 May 2012 10:01:57 +0000

For quering Active Directory /LDAP from command line, you can run something like

curl -u DOMAIN\\\\MYACCOUNT:MYPASSWORD   \\
 "ldap://dc.redaelli.org:3268/OU=users,DC=redaelli,DC=org?memberOf,sAMAccountName?sub?(sAMAccountName=matteo)"

Comments:

Saqib Ali - Dec 1, 2018

Super thanks for posting this!

My first published article SSO Linux Apache Kerberos Active Directory

Sun, 22 Apr 2012 09:14:35 +0000

You can find my article about Single Sign On, Kerberos and Active Directory in the last number of Linux&C

After Oracle also SAP has discovered and integrated R

Sun, 18 Mar 2012 20:06:15 +0000

After Oracle with Oracle R Enterprise, now also SAP has adopted R: SAP HANA provides in-database analytics based on R with its SAP BusinessObjects Predictive Analysis module. For more info about R, please get a look at this R introduction…

Sentiment analysis of Tyre competitors from Twitter and R

Sun, 12 Feb 2012 17:00:19 +0000

Below a simple example of sentiment analyis of tweets about tyre competitors. Notes:

x-axes=positive tweets
y-axes=negative tweets
positive and negative words: Hu and Liu’s “opinion lexicon” categorizes nearly 6,800 words as positive or negative and were downloaded from Bing Liu’s web site: http://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar
only more positive and worst negative tweets are collected

The code can be found here.

Happy new year 2012 with R

Sat, 24 Dec 2011 18:07:55 +0000

http://yihui.name/en/2011/01/happy-new-year-with-r-2011-fireworks/

The 10 Most Important Open Source Projects of 2011

Mon, 19 Dec 2011 20:13:22 +0000

See original post at linux.com

SAP OpenSource NOSQL MongoDB

Sat, 08 Oct 2011 12:03:29 +0000

MongoDB Selected as the Core Content Management Component of SAP’s Platform-as-a-Service (PaaS) Offering “MongoDB’s Flexibility and Scalability Will Enable SAP to Scale Its Content Management Service on Its PaaS to Meet Customer Demand While Managing Data From Different Applications” Read the full marketWatch article.

Oracle Opensource NoSQL Hadoop R

Sat, 08 Oct 2011 10:01:56 +0000

Oracle’s comprehensive big data strategy includes NoSQL, Hadoop, and R analytics “Oracle’s planned distribution of the open-source R statistical environment will be adapted for use on large-scale data within the Oracle database, rather than on desktops and laptops where analysts typically use the software. Oracle R Enterprise will run existing R applications and it will use the R client directly against data stored in Oracle Database 11g. This will vastly increase scalability, performance, and security, according to Oracle, along with the promise of software support. Oracle will ship the open-source distribution along with Linux. Separate R packages with database-specific extensions for Oracle 11g will be bundled with that database”. Taken from an Informationweek article.

Are you a good programmer?

Sat, 20 Aug 2011 06:24:14 +0000

I have just read a funny article about the kind of good programmers at techiferous.com: the philosopher, the inventor, the conqueror, the problem solver..

Twitter data mining with R: generating wordclouds

Mon, 01 Aug 2011 20:06:48 +0000

This image is a sample Word cloud about tweets related to the word linux. It was generated with R (see source code twitter-r-utils) following the examples from:

Apache Solr the opensource Enterprise Search Engine

Fri, 15 Jul 2011 20:56:11 +0000

Aboliamo le province, più potere alle regioni!

Sun, 10 Jul 2011 18:39:07 +0000

Riduciamo i costi della politica italiana abolendo le province e dando alle regioni le competenze che oggi spettano alle province! Di cosa si occupano le province oltre che manutenere le strade provinciali?

Moodle 2.1 released!

Thu, 07 Jul 2011 19:26:08 +0000

“Moodle is a software package for producing Internet-based courses and web sites. It is a global development project designed to support a social constructionist framework of education. Moodle is provided freely as Open Source software (under the GNU Public License).” [from About_Moodle] “You can read more on the Moodle 2.1 release notes, but the major new features are:

A whole new question engine (used by quizzes, for example), which makes questions more robust and will enable developers to create all kinds of interesting question types in the future. Some interesting new features like Certainty-Based Marking have also been added. Huge thanks are due to Tim Hunt and Open University for all their work on this.
Restoring of Moodle 1.9 backups is now possible! Thanks to Moodlerooms for their help in some of the key parts of the core work and to the HQ team for the rest. This was a very complex thing to build due to the many changes between 1.9 and 2.x. Note that only course content in core modules is restored: if you use other activity modules then you need to ask (or help) the developers of those modules to add the new module restore code (thanks to the design this is not a huge task to implement).
Moodle 2.1 now directly supports mobile apps for some functions. The new Moodle app for iPhone (by Moodle HQ) will be released soon (followed by an Android version later) and these require the secure web services of Moodle 2.1 or later. In addition, there is direct support for mobile-specific Moodle Themes and automatic detection of mobile browsers.
Moodle 2.0 Release notes

There are many other smaller improvements too, but one of the ones I really like is the new and improved configuration interface for blocks. You should find it a lot easier to understand than the one in Moodle 2.0.” [from Moodle News]

Digital Identity Mapping: where are you?

Sat, 11 Jun 2011 06:13:07 +0000

[caption id="" align=“aligncenter” width=“500” caption=“digital identity mapping web 2.0”][/caption]

Al via [R]innova di Rante sullo Strategico

Thu, 09 Jun 2011 18:32:01 +0000

"[R]innova vuole essere la prima esperienza di programmazione condivisa per la Web Application Strategico, un progetto open source per l’analisi di serie storiche. L’obiettivo di questa prima edizione di [R]innova è lo sviluppo del layout grafico dei risultati prodotti da Strategico…. Si tratta di un concorso con in palio un …" Leggi l’articolo sul sito rante.org See results of users’s tests

Acqua del rubinetto o in bottiglia?

Wed, 08 Jun 2011 17:43:47 +0000

Guardate questo video.. [youtube]http://www.youtube.com/watch?v=3KpbWAAi7n0[/youtube]

Google Predictor API - release 1.2

Thu, 19 May 2011 19:03:32 +0000

The Google Predictor API V1.2 is out! “The Prediction API provides pattern-matching and machine learning capabilities. Given a set of data examples to train against, you can create applications that can perform the following tasks:

Given a user’s past viewing habits, predict what other movies or products a user might like.
Categorize emails as spam or non-spam.
Analyze posted comments about your product to determine whether they have a positive or negative tone.
Guess how much a user might spend on a given day, given his spending history.

" [see more] There are libraries and sample code for Bash, Java, .Net, Python, Ruby, … For details of samples look at guide Links:

Ford & Google predictor (also here)
Ford & Google Trends

As alternative, for Long Term Prediction over Time Series, you can have a look at Strategico.

Natale 2010: riflessioni di un 40enne depresso

Fri, 24 Dec 2010 18:28:30 +0000

Possibile che nel 2010 ci siano ancora persone che muoiono di freddo a Milano nella quasi totale indifferenza di tutti noi, delle istituzioni e della Chiesa? Ma forse e’ meglio non pensarci e abbuffarsi nei cenoni e pranzi del Santo Natale.

[youtube]http://www.youtube.com/watch?v=Gup4hQnFWyA[/youtube]

Esempi di utilizzo dei dati e mappe Istat con R

Mon, 20 Dec 2010 20:27:13 +0000

[youtube]http://www.youtube.com/watch?v=cUyI6PT7QT4[/youtube] Trovate su R-Istat alcuni semplici esempi di utilizzo con R dei dati e mappe cartografiche forniti liberamente dall’Istat. In questo post un video e un grafico di esempio.. .

R at Google

Mon, 13 Dec 2010 07:47:45 +0000

“…One interesting application is the Google Flu Trends project, which uses R to estimate current flu activity based on Google search results. Google Trends aggregates user search queries showing how often a particular word or phrase has been searched. Correlation tests are run on the search results to obtain a manageable data set of potentially relevant variables. Then using R, they massage the data and create models with optimized weights for each search term. From this, they are able to reasonably estimate current flu activity for different regions around the world. Google also announced an R client for the Google Prediction API (a service which accesses Google’s machine learning algorithms to analyze historic data and predict future outcomes). The R client is available here: http://code.google.com/p/google-prediction-api-r-client/ Final note, Google has published an R Style Guide which may be of interest for those seeking a set of standards for R coding: http://google-styleguide.googlecode.com/svn/trunk/google-r-style.html" Read all the article here!

Rante, il sito degli utenti R in Italia

Wed, 15 Sep 2010 19:40:46 +0000

“Rante è una comunità di utenti appassionati di R, un ambiente di programmazione open source ideato principalmente per l’analisi statistica dei dati. Si fonda nella condivisione della filosofia del software libero e nella convinzione che avere un punto di ritrovo e scambio comune, non solo virtuale, possa portare benefici per tutti coloro che sono interessati all’uso di questo ambiente di programmazione…” [rante.org]

Geographic maps in R: Italy

Mon, 13 Sep 2010 19:09:10 +0000

As suggested by this post, I created an Italian map

with the following R code using the sp package and the GADM maps

library(sp)

con <- url(“http://gadm.org/data/rda/ITA_adm1.RData”)

print(load(con))

close(con)

# giving random values to the Italian regions (1,2,3,5,1,2,3,4,5,…)

language <- rep(seq(1,5),4)

gadm$language <- as.factor(language)

# choosing colors

col = rainbow(length(levels(gadm$language)))

spplot(gadm, “language”, col.regions=col, main=“Italian Regions”)

E io mi lamento...

Sat, 28 Aug 2010 12:29:28 +0000

[youtube]http://www.youtube.com/watch?v=lBB8G3Fc6Rk[/youtube]

Debian celebrates its 17th birthday

Tue, 24 Aug 2010 11:20:46 +0000

Debian celebrates its 17th birthay! Read the official news..

Calling soap web services with curl from a unix shell

Fri, 18 Jun 2010 07:41:17 +0000

[caption id="" align=“alignleft” width=“471” caption=“Curl”][/caption] How to call soap web services from a unix shell? it is easy with curl and, optionally, xsltproc ! Read the tutorial soap requests with curl !

Comments:

Alias Canterbury - Jul 5, 2010

Thanks for posting this tutorial! I really appreciate it, considering that a lot of other websites posted vaguely similar instructions but I couldn’t get them to work.

Thanks for such a nice post. Its a perfect working example. Its a good short guide to get started.

Ebot: an erlang web crawler

Fri, 21 May 2010 17:43:41 +0000

Ebot is a opensource and scalable web crawler written on top of Erlang, a NOSQL database (Apache CouchDB or Riak), RabbitMQ, Mochiweb, Webmachine, .. See my Erlang info page for more details.

RabbitMQ acquired by SpringSource/VMWare

Tue, 13 Apr 2010 18:48:37 +0000

“Erlang-based RabbitMQ was acquired today by SpringSource, which is owned by VMWare” “RabbitMQ is a complete and highly reliable enterprise messaging system based on the emerging AMQP standard. It is licensed under the open source Mozilla Public License and has a platform-neutral distribution, plus platform-specific packages and bundles for easy installation.” Sponsors of AMQP are: Bank of America, N.A., Barclays Bank PLC, Cisco Systems, Credit Suisse, Deutsche Börse Systems, Envoy Technologies Inc., Goldman Sachs, iMatix Corporation sprl., INETCO Systems Limited, JPMorgan Chase Bank Inc. N.A, Microsoft Corporation, Novell, Progress Software, Rabbit Technologies Ltd., Red Hat Inc., Solace Systems Inc., Tervela Inc., TWIST Process Innovations Ltd, WS02 Inc. and 29West Inc… (see here) Go here or here for more details…

The Interactive Internet is quite near... the new facebook chat with XMPP...

Sat, 13 Feb 2010 15:20:22 +0000

After Google with GTalk (~ 180 million users) , now also Facebook (> 400 million users ) has switched its chat protocol to Jabber/XMPP ( pdf introduction). Now you need simply a generic xmpp client (like Adium, iChat, Miranda, Pidgin) and configure Facebook, Gtalk,.. XMPP is now the de facto the Standard protocol for Instant Messaging We should not use anymore chat systems/operators that use proprietary standards like Skype, MS Messanger and Communicator, .. We are next to have an Interactive Internet: like emails that can be sent out of a domain, we’ll be able to chat with people outside our domain: with my gmail account I’ll be able to chat with facebook people !!!!!!!!!!!! The new facebook chat (chat.facebook.com) runs an OpenSource software witten in Erlang, ejabberd (commercial support at process-one.net). That will be also used by the next moodle 2.0. See the video “Set-up an Enterprise Instant Messaging server in 180 seconds”

Comments:

matteo - Feb 6, 2010

Is Yahoo also experimenting XMPP? Look at http://www.process-one.net/en/blogs/article/after_aol_yahoo_is_also_experimenting_with_xmpp

Do not forget...

Sun, 31 Jan 2010 07:39:57 +0000

Sometimes I need to watch this video, it helps me to not forget…

[youtube]http://www.youtube.com/watch?v=8hOaL6M4SDU[/youtube]

QUESTO SUCCEDE MENTRE NOI SIAMO ACCECATI DALL’AVIDITA’ E DAL CONSUMISMO…

E NOI FACCIAMO poco o NULLA PER CAMBIARE…

Questo video e’ la mia bussola…

A new blog for me!

Tue, 05 Jan 2010 18:31:11 +0000

My blog now lives at redaelli.org: new posts will be published automatically also in my facebook and twitter profiles. But I’m looking also at blogger: I’m playing with http://redaellicarate.blogspot.com/

Do NOT BUY any DVB USB devices with TRIDENT chipsets

Fri, 01 Jan 2010 20:04:18 +0000

Comments:

Lincoln B. Marcelo - Jan 0, 2011

Of course I want sources to make my own Delphi/Kylix program to control my tv adaptor box. It has tm5600 and I want to write my own controller. I bought a ENUTV-2 USB TV box this month (Dec 2010) and I think I have rights as user and owner of a device that has this chip inside to write computer controls to use it as i want. I need an linux driver to make it work under SuSE, I need the commands to make my Pascal or C++ personal controller for it.

Happy new year 2010

Wed, 30 Dec 2009 10:52:32 +0000

I hope 2010 will be a better year for all people in the world…

BRING THE BOYS BACK HOME!!!

Tue, 01 Dec 2009 18:29:12 +0000

[youtube=http://www.youtube.com/watch?v=1_pszQM6AVs]

..ma sI continua ad inviare nuove truppe in una guerra scatenata da prove riconosciute INFONDATE !!!!!!!!!!!!

Learning Erlang

Sat, 31 Oct 2009 10:40:09 +0000

Apache CouchDB ~ Ubuntu

Sat, 31 Oct 2009 10:14:47 +0000

"Ubuntu 9.10 Karmic Koala has just been released. This is big news as this version includes Apache CouchDB, used as a replicable database by desktop apps. This means CouchDB will be on over 10 million desktops" [from Damien Katz] Erlang tutorial

oRestE: a REST interface to your databases with Erlang

Wed, 07 Oct 2009 07:24:52 +0000

oRestE gives you a fast way to expose your database with a restful interface oRestE is built on top of ERLANG (Introduction), MOCHIWEB and WEBMACHINE (webmachine.pdf) oRestE it is light, stable and fast! thanks to ERLANG/OTP and MOCHIWEB

Some Reasons for Loving Google Apps

Fri, 25 Sep 2009 11:22:38 +0000

A quick presentation about why people & employees & IT managers & companies should love and adopt Google Apps instead of other old style Intranets (MS Sharepoint, Sap Portal, .. that require the (not user friendly) programmers’s actions checkin/checkout) and Email Suites (MS Exchange)

Ewg: Erlang Wordlist Generator

Tue, 04 Aug 2009 20:01:10 +0000

Ewg is a Wordlist generator written in Erlang. Thanks to this simple project I have been improving my knowledge in Erlang/OTP and Git. I love functional programming and learning more and more… ;-)

Erlang at Facebook, SAP, ...

Fri, 26 Jun 2009 09:45:03 +0000

Erlang At? Please add a comment if you use it in your company! Erlang at BBC

http://www.erlang-factory.com/upload/presentations/147/EndaFarrell-ErlangFactoryLondon2009-ErlangattheBBC.pdf

Erlang at Facebook

Erlang at Github

http://github.com/blog/112-supercharged-git-daemon

Erlang at SAP

http://www.erlang-factory.com/upload/presentations/57/SumeetBajaj_ErlangatSAP.pdf

Erlang at 37Signals

http://beebole.com/en/blog/erlang/37signals-dives-into-erlang/

Comments:

The Interactive Internet is quite near… the new facebook chat with XMPP… « Matteo Redaelli - Feb 6, 2010

[…] new facebook chat uses an OpenSource software witten in Erlang, ejabberd!! That will be also used by next moodle 2.0 chat […]

http://stackoverflow.com/questions/1636455/where-is-erlang-used-and-why

Facebook.com Architecture

Fri, 26 Jun 2009 06:55:24 +0000

Do U want to learn something about the architecture of Facebook.Com? See this presentation

Connecting to a database with a webService (cxf) deployed (osgi) in Servicemix

Sun, 07 Jun 2009 17:37:15 +0000

Install and start ServiceMix 4.0 2) Add necessary bundles

features/install cxf-osgi

osgi/install -s mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.commons-dbcp/1.2.2_3/

osgi/install -s wrap:mvn:org.springframework/spring-jdbc/2.5.6/

osgi/install -s wrap:mvn:mysql/mysql-connector-java/5.1.6

… See details in this gooole document or download test-wsdl-first-osgi.zip

Comments:

Thomas Gill - Aug 1, 2009

Matteo - How do you install an Oracle driver in Servicemix. I have the oracle6.jar. Tried putting in lib dir and also install file:///tmp/ojdbc6.jar Trying to setup Master -Slave using JDBC for ActiveMQ

In smx4 try installing bundle from springsource http://www.springsource.com/repository/app/bundle/version/detail?name=com.springsource.oracle.jdbc&version=10.2.0.2 if you cannot, please ask in servicemix forum, I’m not an expert in java/osgi.. Regards

Thanks Matteo - The Spring version is a little old but works. I have the ActiveMQ tables created in DB and everything works. I tried with one Master alone and when running it is all good. But when the Master goes down the DB lock is not removed. Planning to start 2 Brokers and see how it behaves.

Howto sharing (osgi style) a database connection in Servicemix4 / FuseEsb4

Sun, 07 Jun 2009 12:58:58 +0000

ISTAT is moving from SAS to R

Fri, 05 Jun 2009 14:49:43 +0000

“Istat [..] helps to develop ‘R Project’, a statistical open source application. “We have donated software libraries to R, and are moving away from using SAS, the proprietary alternative to R. We contributed to the statistical application Adamsoft, which is being developed at the Caspur computing-lab, just two kilometres away from our institute.” Vaccari says adopting open source requires institutes to make adjustments in their organisational culture. “It changes how people work, from ‘I’m important because I’m a bottleneck, to I’m important because I share things’.” Read it!

Connecting to a database with a webService (cxf) deployed (jbi) in Servicemix

Sat, 30 May 2009 10:53:58 +0000

Introduction

Apache ServiceMix at the moment doesn’t have a JBI component for connecting to a database (servicemiix-jdbc is in the roadmap): in the meanwhile you have to use servicemix-cxf or servicemix-bean. This example (cxf-wsdl-first-jdbc.zip, cxf-wsdl-first.pdf) is built on top of the example cxf-wsdl-first and can be deployed in apache servicemix 4.0 as JBI sa package.

Installing smx4

Download from “http://servicemix.apache.org/SMX4/download.html”

Unzip it

cd /usr/local

tar xvfz apache-servicemix.tar.gz

Starting smx4

cd /usr/local/apache-servicemix-4.0.0/

./bin/servicemix

Creating WS

cd /usr/local/apache-servicemix-4.0.0/examples

cp -a cxf-wsdl-first cxf-wsdl-first-jdbc

Edit file “wsdl-first-cxfse-su/pom.xml”

mysql

mysql-connector-java

5.0.8

Edit file “wsdl-first-cxfse-su/src/main/resources/xbean.xml”

class=“org.springframework.jdbc.datasource.DriverManagerDataSource”>

value="jdbc:mysql://localhost:3306/moodle" />

Edit file “wsdl-first-cxfse-su/src/main/java/org/apache/servicemix/samples/wsdl_first/PersonImpl.java”

import java.util.\*;
import javax.sql.DataSource;
import org.springframework.jdbc.core.JdbcTemplate;

private DataSource dataSource;

private JdbcTemplate jdbcTemplate;

public void setDataSource(DataSource dataSource) {

this.dataSource = dataSource; this.jdbcTemplate = new JdbcTemplate(dataSource); }

public void getPerson(Holder personId, Holder ssn, Holder name)

throws UnknownPersonFault

org.apache.servicemix.samples.wsdl_first.types.UnknownPersonFault fault = new org.apache.servicemix.samples.wsdl_first.types.UnknownPersonFault();

fault.setPersonId(personId.value);

if (personId.value == null || personId.value.length() == 0) {

   throw new UnknownPersonFault(null, fault);

}

List result = this.jdbcTemplate.queryForList("select description, lastname from mdl\_user where username = ?",
   new Object\[\]{personId.value});

if(result.size() != 1) throw new UnknownPersonFault(null, fault);

 Map record=(Map)result.get(0);
 name.value = record.get("lastname").toString();
 ssn.value = record.get("description").toString();
}
```

Deplying the WS
===============

mvn clean instal cp wsdl-first-cxf-sa/target/wsdl-first-cxf-sa-4.0.0.zip /usr/local/apache-servicemix-4.0.0/deploy/

Testing WS
==========

Open with your browser the file “file:///usr/local/apache-servicemix-4.0.0/examples/cxf-wsdl-first-jdbc/client.html”

Http statistics with curl

Wed, 15 Apr 2009 15:49:43 +0000

With CURL and RRDTOOL it is possible to collect http request statistics. Below you can find the 2 scripts #************************ #curl-rrdtool.sh #************************ #!/bin/bash # Paths… adjust as necessary BASE="/home/matteo/curl" CONFIG=“config” CURL="/usr/bin/curl" RRDTOOL="/usr/bin/rrdtool" # Given a URL, return a file name # ensure we have the needed parameters # Strip out any non alphas, replace with underscores, # and add a host name FILE=apache.rrd # Can only return numeric results, so global $FILE # has the result # Called to create the RRD create_rrd () { # ensure we have the needed parameters # Convert the URL to a name and build the RRD $RRDTOOL create “$BASE/$FILE” -s 30 \ DS:total:GAUGE:60:0:U DS:dns:GAUGE:60:0:U DS:connect:GAUGE:60:0:U \ DS:pretransfer:GAUGE:60:0:U DS:starttransfer:GAUGE:60:0:U \ RRA:AVERAGE:0.5:1:288 RRA:AVERAGE:0.5:6:336 RRA:AVERAGE:0.5:24:372 \ RRA:AVERAGE:0.5:144:730 RRA:MIN:0.5:1:288 RRA:MIN:0.5:6:336: \ RRA:MIN:0.5:24:372 RRA:MIN:0.5:144:730 RRA:MAX:0.5:1:288 \ RRA:MAX:0.5:6:336 RRA:MAX:0.5:24:372 RRA:MAX:0.5:144:730 \ RRA:LAST:0.5:1:288 } if [ ! -f “$BASE/$FILE” ]; then create_rrd $URL fi while true ; do $CURL -b /tmp/cookies.txt -c /tmp/cookies.txt -d “username=test1&password=XXXXX” -s http://learninglab.group.mysite.com/moodle/login/index.php OUT=`$CURL -b /tmp/cookies.txt -c /tmp/cookies.txt -m 30 -w %{time_total}:%{time_namelookup}:%{time_connect}:\ %{time_pretransfer}:%{time_starttransfer} -o /dev/null -s http://learninglab.group.mysite.com/moodle/` $RRDTOOL update “$BASE/$FILE” -t “total:dns:connect:pretransfer:starttransfer” N:$OUT sleep 30 done #************************ #curl-rrdtool-chart.sh #************************ #!/bin/sh MEASUREMENT=“matteo” SITE=“http://learninglab.group.mysite/moodle/" FILE=/home/matteo/curl/apache DEFS=” DEF:total=$FILE.rrd:total:AVERAGE \ DEF:dns=$FILE.rrd:dns:AVERAGE \ DEF:connect=$FILE.rrd:connect:AVERAGE \ DEF:pretransfer=$FILE.rrd:pretransfer:AVERAGE \ DEF:starttransfer=$FILE.rrd:starttransfer:AVERAGE" width=800 height=400 rrdtool graph $FILE-hourly.png \ –width $width –height $height \ -r -s end-30min \ -t “Performance of $SITE from $MEASUREMENT” -v “Seconds” \ $DEFS \ AREA:total#FF0000:“Data Transfer” \ AREA:starttransfer#CAAA00:“First Byte” \ AREA:connect#0000FF:“Connection” \ AREA:dns#00FF00:“DNS Lookup” \ COMMENT:"\n" COMMENT:"\n" \ GPRINT:total:AVERAGE:“Average request time %0.2lf %Ss” \ GPRINT:total:MIN:“Min/Max %0.2lf %Ss/” \ GPRINT:total:MAX:"%0.2lf %Ss" \ COMMENT:"\n" \ GPRINT:total:LAST:“Current request time %0.2lf %Ss”

ICA-ATOM (QUBIT), THE OpenSource archival description software

Wed, 15 Apr 2009 08:51:30 +0000

I’ve collected some information about ICA-ATOM (based on QuBit). Please read my presentation from Google Docs

Ruby Wordlist Generator (ruby-wg)

Mon, 13 Apr 2009 19:44:02 +0000

After many years I’ve updated my Wordlist Generator project! I’ve seen in several forums that some people still use my old perl script wg.pl The new ruby project is hosted at htt p://code.google.com/p/ruby-words-generators/ ruby-wg can be used to generate wordlist for the famous password cracker John the Ripper

Google si unisce alla battaglia della UE contro Internet Explorer

Fri, 27 Feb 2009 17:52:41 +0000

L’Antitrust della Comunita’ Europea continua la sua battaglia contro Microsoft e Google dara’ il suo contributo. Leggete qui'

Microsoft Tax and Apple tax

Thu, 08 Jan 2009 08:12:20 +0000

Microsoft dice che ogni volta che si compra un Mac si paga una tassa ad Apple. Ma si puo’ dire analogamente che ogni volta che si compra un pc con installato windows, si paga una tassa a Microsoft. Con Linux e OpenOffice la stragrande maggioranza degli utenti di computer potrebbero risparmiare la Microsoft tax e per di piu’ non avere piu’ problemi di virus!!!!

Comments:

mika29 - Jan 5, 2009

A parte il fatto che comprando Windows i soldi vanno a Microsoft, consideriamo che quando si compra un Mac parte dei soldi va sempre a Microsoft, che possiede alcune azioni della Apple….

N queens problem with erlang

Mon, 05 Jan 2009 11:41:35 +0000

“The eight queens puzzle is the problem of putting eight chess queens on an 8×8 chessboard such that none of them is able to capture any other using the standard chess queen’s moves. The queens must be placed in such a way that no two queens would be able to attack each other. Thus, a solution requires that no two queens share the same row, column, or diagonal. The eight queens puzzle is an example of the more general n queens puzzle of placing n queens on an n×n chessboard, where solutions exist only for n = 1 or n ≥ 4… “ [Wikipedia] Below my solution using ERLANG (very strong for concurrent programming, used by Ericsson, Facebook, Amazon,Google,..) with List comprehensions and suggestions from other sites

-module(queens).
-export(\[queens/1\]).

same_row({_,X},{_,X}) -> true;
same_row({_,_},{_,_}) -> false.

same_column({X,_},{X,_}) -> true;
same_column({_,_},{_,_}) -> false.

same_diagonal({X1,Y1},{X2,Y2}) ->
	DeltaX = abs(X1 - X2),
	DeltaY = abs(Y1 - Y2),
	DeltaX == DeltaY.

attaccable({X1,Y1}, {X2,Y2}) ->
	same_row( {X1,Y1}, {X2,Y2} ) orelse
	same_column( {X1,Y1}, {X2,Y2} ) orelse
	same_diagonal( {X1,Y1}, {X2,Y2} ).

safe_cell({_,_}, []) -> true;
safe_cell({X1,Y1}, [{X2,Y2}|L]) ->
	not attaccable({X1,Y1}, {X2,Y2}) andalso
safe_cell({X1,Y1}, L).

queens(N) ->
	S = queens(N, N),
	io:format("~w solutions found!", [length(S)]),
	S.
queens(0, _) -> [[]];
queens(X, Rows) ->
	[[{X,Y}|L] ||
	L <- queens( X - 1, Rows),
	Y <- lists:seq(1,Rows),
	safe_cell({X,Y}, L)
	].

Basta Portali Intranet Unidirezionali!

Sat, 27 Dec 2008 13:19:47 +0000

Lavoro per un’Azienda che, come altre di vecchio stampo, persiste con un modello di Intranet unidirezionale (dall’alto verso il basso dove il dipendente e’ passivo) che non stimola e incentiva la collaborazione e la condivisione del know-how. Purtroppo e’ una questione di mentalita’ e non di strumenti. Strumenti validi, robusti, gratuiti, e OpenSource ce ne sono… Leggete qui’. http://www.slideshare.net/matteoredaelli/wordpress-mu-presentation http://www.slideshare.net/folletto/wordpress-as-cms http://www.slideshare.net/msincome/how-to-use-wordpress-widgets http://www.slideshare.net/msincome/wordpress-css-editing http://www.slideshare.net/bazza/high-performance-wordpress

Cambio programma

Fri, 21 Nov 2008 08:26:10 +0000

Oggi mi sono svegliato presto, mi sono infilato i pantaloni, vestito lentamente, preparato caffè, preso le mie mazze da golf e sono andato piano verso il garage, ho messo le mazze nell’auto e ho tirato fuori la macchina dal garage sotto una pioggia torrenziale. La strada era totalmente inondata e il vento gelido soffiava a 100 km orari. Sono rientrato con l’auto in garage, acceso la radio e sentito che le previsioni dicevano che quel tempaccio sarebbe durato tutto il giorno. Che delusione! Sono rientrato in casa, mi sono rispogliato e silenziosamente sono scivolato a letto. Piano mi sono avvicinando a mia moglie stringendola e, mettendole una mano sulla natica, le ho sussurrato all’orecchio: “il tempo fuori è orribile” Lei mezza addormentata mi ha risposto: “Sì lo so, e tu pensa a quel cretino di mio marito che è andato a giocare a golf!”

Come contribuire alla ricerca donando qualche ora del tuo PC

Wed, 19 Nov 2008 22:15:54 +0000

E’ facile. Scarica un piccolo programma (boinc) e installalo sul tuo computer. Nei tempi morti o comunque quando vuoi tu, il programma si collega ai centri di ricerca, scarica un job da elaborare e quando ha terminato reinvia i risultati scaicando poi un nuovo job…. World Community Grid e’ un esempio e si occupa di progetti “that hold tremendous potential to benefit humanity.

Nutritious Rice for the World
Help Conquer Cancer
Discovering Dengue Drugs - Together
Human Proteome Folding - Phase 2 Project
FightAIDS@Home Project
How Grid Computing Works

Io personalmente mi sono iscritto al Team di Oses.

Calendario 2009

Sat, 18 Oct 2008 11:48:44 +0000

[slideshow id=792633534436779082&w=426&h=320]

Alro che calendario Pirelli!! Ecco il Calendario 2009 di Binario per l’Africa con le foto scatatte in Kenia l’agostro scorso.

Here is the African Calendar 2009!

Using Apache Camel with Ibatis

Wed, 08 Oct 2008 20:54:26 +0000

This is a quick tutorial about the usage of the very powerful Integration tool Apache Camel with Apache Ibatis. Many thanks to Apache Camel forums and Claus Ibsen.

REQUIREMENTS

I used Java version 1.5.0_16
Apache Maven version 2.0.9
A database mysql: I used a local db “moodle”, user “moodle”, password “moodle” and a table “mdl_user” [ create table mdl_user ( id int, firstname varchar(30), lastname varchar(30), email varchar(30) ); ]

DOWNLOAD

The full sample project can be download from here!

STEP BY STEP

As suggested by http://activemq.apache.org/camel/creating-a-new-spring-based-camel-route.html matteo@nowar:~/workspace$ mvn archetype:create \ -DarchetypeGroupId=org.apache.camel \ -DarchetypeArtifactId=camel-router \ -DarchetypeVersion=1.4.0 \ -DgroupId=org.redaelli \ -DartifactId=demo-ibatis matteo@nowar:~/workspace$ cd demo-ibatis matteo@nowar:~/workspace$ mvn eclipse:eclipse matteo@nowar:~/workspace$ mvn install Create iBatis files

src/main/resources/SqlMapConfig.xml
src/main/resources/User.xml
src/main/java/org/redaelli/User.java

Add the routes and dependences in the files

src/main/java/org/redaelli/MyRouteBuilder.java
pom.xml (adding dependencies “camel-ibatis” e “mysql jdbc”)

And finally run the demo with matteo@nowar:~/workspace$ mvn camel:run

The site of St Massimo Parish del Meru Kenya is online

Mon, 25 Aug 2008 08:31:40 +0000

Please, go and visit the new website of St Massimo Parish. I hope it will be reacher and reacher of contents and useful information…

Alcune verita' sugli inceneritori...

Fri, 01 Aug 2008 12:37:41 +0000

Mi e’ stata girata per email questa presentazione, non so quanto corrisponda al vero ma e’ di sicuro un interessante spunto di riflessione… Incenerire inquina e… UCCIDE?

Set a Guinness World Record Enjoy a Better Web

Fri, 13 Jun 2008 17:03:58 +0000

Pledge to get Firefox 3 during Download Day to set the Guinness World Record for Most Software Downloaded in 24 Hours!

OpenID Demand: please go and visit the site...

Fri, 23 May 2008 18:21:03 +0000

**I’m tired of having and remembering so many userid/password for internet sites, aren’t you? ** Please, o and visit the site “OpenID Demand”.. http://demand.openid.net

Presentazione Free e Opensource Software (e Linux)

Mon, 05 May 2008 14:35:16 +0000

Trovate qui’ una presentazione sul Free & Opensource sw che ho fatto. Ho tagliato alcune slide sulla mia azienda per evitare eventuali problemi di sicurezza.

Assegno

Wed, 30 Apr 2008 11:43:40 +0000

C’era un uomo che aveva lavorato tutta la vita, aveva risparmiato tutti i suoi soldi, e quando si trattava di spenderli era un vero avaro. Poco prima di morire, disse alla moglie… “Quando muoio, voglio che tu prenda tutti i miei soldi e li metta nella bara con me. Me li voglio portare con me nell’aldilà.” E così si fece promettere con tutto il cuore dalla moglie, che quando sarebbe morto lei avrebbe messo tutti i suoi soldi nella cassa con lui. Al funerale, era steso nella bara con vicino la moglie, vestita di nero, seduta affianco alla sua migliore amica. Quando fu finita la cerimonia, e si preparavano a chiudere la bara, la moglie diss: “Aspettate un momento!” Aveva una piccola scatola di metallo; si avvicinò con la scatola e la mise nella cassa. Chiusero la bara e la portarono via. E quindi la sua amica le disse: “Ragazza, sapevo che non eri così tonta da mettere tutto quel denaro la dentro con tuo marito.” La mogie fedele rispose: “Senti, io sono una persona credente; non posso tornare sulle mie parole. Gli ho promesso che avrei messo quei soldi nella bara con lui.” “Vuoi dire che hai messo tutto quel denaro li dentro con lui!?!?!?” “Certo che l’ho fatto”, disse la moglie. “L’ho preso tutto, l’ho messo sul mio conto, e gli ho fatto un assegno……. Se riesce a incassarlo se li può spendere tutti.”

Test your web site with all browsers & operating systems

Wed, 09 Apr 2008 15:23:21 +0000

Go and see the web site http://browsershots.org/

Boinc

Tue, 01 Apr 2008 06:33:56 +0000

Use the idle time on your computer (Windows, Mac, or Linux) to cure diseases, study global warming, discover pulsars, and do many other types of scientific research. It’s safe, secure, and easy with boinc

Comments:

tramp - Apr 3, 2008

The idea is surely praiseworthy and I was about to subscribe. But suddenly I stopped and some strange thoughts started to surf into my mind. Why should I give my free help to a foreign university / department? I would be happier if I could give this same contribution to an italian organization. Maybe I’m wrong and I’m a bit selfish, but I think: “what would those people do with the results of their studies?”; “will they make all the other peolple pay for that?”. So I say “NO, I won’t subscribe!”

Amnesty International & Tibet

Thu, 20 Mar 2008 18:28:38 +0000

…a seguito delle proteste scoppiate lunedì 10 marzo in Tibet in cui sono stati arrestati numerosi monaci tibetani, Amnesty International ha lanciato oggi un’azione urgente in favore di 15 monaci arrestati e scomparsi. Si tratta di Samten, Trulku Tenpa Rigsang, Gelek Pel, Lobsang, Lobsang Thukjey, Tsultrim Palden, Lobsher, Phurden, Thupdon, Lobsang Ngodup, Lodoe, Thupwang, Pema Garwang, Tsegyam e Soepa, in carcere dal 10 marzo per aver preso parte a una manifestazione pacifica a Barkhor, Lhasa, la capitale della Regione autonoma tibetana. Non si hanno ulteriori informazioni sul luogo in cui sono detenuti né su eventuali accuse formulate nei loro confronti e Amnesty International teme rischino di subire torture e altri maltrattamenti. Perchè non firmare e far firmare l’appello per la loro scarcerazione? Valli a questa pagina !

Detrazione per gli abbonamenti ai mezzi di trasporto pubblici

Wed, 19 Mar 2008 08:38:42 +0000

Finalmente la detrazione per gli abbonamenti ai mezzi di trasporto pubblici: art. 1, comma 309, legge n. 244 del 27 dicembre 2007. Ecco il testo del rimborso_abbonamenti.pdf!

E' la fine della storia

Mon, 25 Feb 2008 19:18:22 +0000

“E’ la fine della storia, e non lo sai. Lui e’ li’, in piedi davanti alla finestra, e tu non gli perdoni di schermare la luce. Non e’ lui che vedi, ma il giorno, cui lui impedisce di entrare. Inizia cosi” [Brigitte Giraud], tratto dal quotidiano City, 25-02-2008

ATM: collegamento Linate con le stazioni FS o fermate della metropolitana

Mon, 25 Feb 2008 09:51:39 +0000

ATM, nella sua pagina dedicata sul quotidiano gratuito City, oggi ha fatto pubblicita’ all’UTILE servizio di collegamento tra Linate e San Babila fornito dalla linea 73. La considero una presa in giro e un’offesa per tutti i cittadini milanesi e lombardi! E una concessione alla lobby dei tassisti… La maggior parte di chi parte/arriva a Linate abita in centro a Milano? Oppure quanti hanno la NECESSITA’ di passare per il centro della citta’ con le valigie prima o dopo il volo? Quello di cui abbiamo bisogno e’ un collegamento tra Linate e le stazioni FS (tipo Lambrate e Rogoredo) e del metro’ (tipo Gobba) della citta’. Vergogna!!!!!!!!

Foto della ciaspolata a S. Caterina con gli Amici della Montagna

Mon, 04 Feb 2008 12:09:54 +0000

In questa pagina trovate le foto della ciaspolata di ieri a S. Caterina organizzata dagli Amici della Montagna (Alpini di Carate Brianza)

Che vergogna...

Fri, 25 Jan 2008 19:17:27 +0000

I deputati sono i rappresentanti del nostro Paese e dovrebbero dare il buon esempio. Quanto successo al Senato ieri prima e dopo l’annuncio della sfiducia al Governo e’ pietoso… E’ utopia ma sarebbe bello varare prima delle elezioni delle leggi che impediscano la candidatura alle persone indagate, condannate o che abbiamo gia’ partecipato almeno due legislature!!! I deputati prendono TROPPI soldi, molti lo fanno SOLO per soldi e interessi personali, non per AMORE per il Nostro Paese…

Comments:

superbolo - Jan 0, 2008

Tutti i politici.. da chi sta in alto fino alla fine della filiera nei comuni … per Esempio, Cuffaro, perchè pensi si sia dimesso? Per onestà? No … perchè, una volta caduto il governo, il consiglio dei ministri non puo’ piu’ firmare il suo allontanamento dal parlamento regionale, e con le nuove elezioni, Toto, potrà diventare senatore e godere dell’immunità parlamentare….

E' morto Bobby Fischer!

Fri, 18 Jan 2008 22:22:17 +0000

All’eta’ di 64 anni e’ morto il mitico scacchista Bobby Fischer divenuto campione nel mondo nel 1972! Links:

* http://bobbyfischer.net/

* http://www.bobby-fischer.net/

One laptop for a child

Thu, 27 Dec 2007 19:23:27 +0000

Per maggiori info andare sui siti

Linux in the slum

Sun, 11 Nov 2007 07:59:48 +0000

AfrikaSi ha completato la prima fase del suo progetto pilota “Linux in the slum”, iniziativa volta a promuovere l’alfabetizzazione informatica dei giovani che vivono nelle baraccopoli di Nairobi. Questa iniziativa fa parte di un progetto piu’ ampio: Linux Aids People! Leggi il resto dell’articolo su Oses.it. Altre info su cissiboy linux-e-le-speranze-africane

Linux Aids People

Sun, 11 Nov 2007 07:29:50 +0000

Nell’ambito delle attività solidali di lotta al sottosviluppo, alla povertà, al disagio, OSES ha avviato il progetto permanente Linux Aids People che mira alla raccolta e riconversione, con soluzioni Open Source, di Personal Computer da destinare ad uso didattico nelle scuole dei paesi in via di sviluppo.

Comments:

Federico - Nov 4, 2007

Ciao Matteo, mi sono iscritto sul sito di OsES. Potrei sapere come collaborare? Grazie Federico

SAP Contributes New Tool to Open Source Eclipse Developer Community

Tue, 06 Nov 2007 20:04:39 +0000

Advanced Memory Analyzer Capabilities Enable Eclipse Developers to Build Enterprise Applications Within Leading Open Source Development Environment Read the rest f the article at CNN. Sap Wiki at https://wiki.sdn.sap.com/wiki/x/k2w

Avete mai provato OpenOffice?

Wed, 03 Oct 2007 14:36:55 +0000

OpenOffice e’ una suite office GRATUITA scaricabile liberamente da internet che legge e scrive documenti di Microsoft Office (doc,xls,ppt,..). Stupisce per le similarita’ di aspetto, funzionalita’ e usabilita’ col suo concorrente commerciale…Su http://www.linux.com/feature/119513 trovate una comparazione tra le due suite circa la creazione di presentazioni… Io da anni uso unicamente OpenOffice per leggere e scrivere documenti di lavoro e uso spesso la funzione di export as PDF…

E' notte per l'ultima volta

Fri, 14 Sep 2007 06:28:08 +0000

_E’ notte per l’ultima volta._Per l’ultima volta le tue mani si raccolgono sul mio corpo. Domani sara’ autunno. Insieme seduti in terrazza guarderemo le foglie secche spargersi sul villaggio come lettere che bruciano, una per una, in case separate.

Louise Gluck, quotidiano CITY

CV online

Wed, 12 Sep 2007 09:54:21 +0000

Ciao Ho messo online il mio curriculum vitae. Se cercate qualcuno che aiuti la vostra Azienda (a Milano, Lecco o Brianza) a traghettare verso un’informatizzazione libera e + economica (grazie a Linux e ai programmi OpenSource) io potrei essere la persona giusta per voi… http://matteoredaelli.wordpress.com/cv/

OpenID for SSO

Sat, 08 Sep 2007 06:11:56 +0000

OpenID has been getting more and more famous and many OpenSource projects have already adopted it … http://openid.net/presentations.bml http://www.openidbook.com/chapters/OpenIDBook-Draft-13.pdf There are libraries for the most common programming languages (Python, Ruby, Java, Perl, .Net) http://openid.net/wiki/index.php/Libraries How does it work? Get an OpenID userid from an OpenID provider (wordpress.com, aol, livejournal,.. OpenIDServers) or from your own provider OpenID (see Run_your_own_identity_server, i.e crowd is used by several big companies but you have to pay for it if you use it not for openSource projects) and then you can use that account (if you want) to enter all other OpenID enables sites (https://www.myopenid.com/directory) http://openid.net/pres/2007-kveton-openid-5.pdf

Comments:

Matteo - Jan 4, 2008

Anche Google sceglie OpenID come sistema di Identity e SSO. Vedete http://punto-informatico.it/p.aspx?id=2155653 e http://uk.techcrunch.com/2008/01/09/google-ibm-and-verisign-to-join-openid/ Con la sola userid/password di Google (o di wordpress.com, AOL, o qualsiasi altro provider OpenID) sara’ possibile accedere (in SSO) a tutti i siti OpenID enabled.

Anche sourceforge accetta utenze OpenID! http://openid.net/2008/05/01/sourceforge-allows-openid-logins/

Howto Integrating SAP Netveawer and OpenID http://www.sap.com/mk/get?_EC=aVAixZ7ktAaSgYDKqVnHF6

Cena di beneficenza

Thu, 06 Sep 2007 14:41:38 +0000

Vi propongo la cena di beneficenza sabato 13 ottobre 2007 alle ore 20 presso l’Agriturismo “I LAGHETTI” via Cavamarna (Oasi di Baggero) - Monguzzo (CO) durante la quale verra’ presentato il gruppo di volontariato Binario per l’Africa. Per maggiori informazioni andate sul sito http://binarioperlafrica.wordpress.com/ e se volete ad agosto andare in Kenia nella zona del Meru per vedere con i vostri occhi a cosa servono i contributi raccolti…

Irlanda On the Road

Tue, 07 Aug 2007 06:59:56 +0000

Le due settimane centrali di agosto sono in Irlanda per 2 settimane con Eleonora, Marino e Paola. Nella sezione foto troverete le fotografie fatte da me e Marino. Venerdi’ 10 agosto 2007 Milano - Shannon - Bunratty - Cashel Partenza con Ryanair da Orio al Serio alle 10.40 per Shannon dove abbiamo gia’ noleggiato la macchina. B&B Ballyknock House, Cashel Sabato 11 agosto 2007 Cashel - Kinsale - Bandon B&B Derrycool Domenica 12 agosto 2007 Bandon - Mizen Head - Kenmare B&B Hazelwood, Killaha Lunedi’ 13 agosto 2007 Kenmare - Ring of Kerry - Kenmare Martedi’ 14 agosto 2007 Kenmare - Castelmaine B&B Castelmaine House Mercoledi’ 15 agosto 2007 Castelmain - Dingle - Tarbet - Killimer - Doolin Giovedi’ 16 agosto 2007 Doolin - Cliff of Moher - Doolin Venerdi’ 17 agosto 2007 Doolin - Galway - Spiddle B&B Kelly’s Driftwood Sabato 18 agosto 2007 Spiddle - Connemara - Castlebar. B&B Primrose Cottage (the best one) Domenica 19 agosto 2007 Castebar - Achill Island - Sligo B&B Walrey, Rosses Point Road Lunedi’ 20 agosto 2007 Sligo - Bunberg B&B An Teach Ban, pmcgill@iol.ie Martedi’ 21 agosto 2007 Bunberg - Dunfanaghy beach - Boyle Mercoledi’ 22 agosto 2007 Boyle - Ennis B&B Brookville (the worst and expensive one) Giovedi’ 23 agosto 2007 Ennis - Limerick - Killrush - Shannon airport Venerdi’ 24 agosto 2007 Dopo una notte in aeroporto si prende il volo per Orio al Serio alle 6.40 del mattino.

Detassiamo l'abbonamento ai mezzi pubblici!

Fri, 03 Aug 2007 19:00:03 +0000

Lavoro per un noto gruppo industriale a Milano, citta’ decadente a causa del traffico, dell’aria inquinata e l’immobilita’ politico ambientale degli ultimi decenni. Conosco diversi colleghi che pur vivendo in citta’ vengono sistematicamente al lavoro ogni giorno in macchina. Come si potrebbe incentivare i cittadini e pendolari all’utilizzo dei mezzi pubblici? Oltre alla scontato miglioramento della frequenza dei mezzi si potrebbe permettere ai lavoratori di detrarre dalle tasse il costo degli abbonamenti ATM e Trenitalia usati per recarsi al lavoro. Che ne dite? VI sembra un’idea troppo insensata o irrealizzabile?!?!?!

Comments:

superbolo - Aug 2, 2007

I mezzi di trasporto in Italia sono ben lontani dall’essere efficenti, puliti e moderni. E quando lo sono, per colpa dei soliti, e quasi impossibile utilizzarli. E poi ci sono gli orari, quasi sempre studiati da chi non ha reale necessità di prenderli. Certo, è facile fare critica, ma purtroppo è anche troppo difficile ottenere qualcosa. E anche per chi utilizza le due ruote per recarsi al lavoro, non è facile e non incentivato. Ecco quindi che tanti preferiscono rintanarsi per ore ed ore in una scatola in coda, pero’ al fresco dell’aria condizionata od al caldo del riscaldamento. Che vita pero’ ….

… a volte arriva qualche buona notizie (anche se c’è da vedere se passerà davvero…). come questa, contenuta nella finanziaria 2008: Trasporto pubblico locale Per le spese, non superiori a 250 euro, sostenute fino al 31 dicembre 2008 per l’acquisto di abbonamenti ai servizi di trasporto pubblico locale, regionale e interregionale, è prevista una detrazione del 19 per cento. Per poterne usufruirne è necessario che le spese non siano deducibili dai singoli redditi che concorrono a formare quello complessivo. chissà! speriamo :)

Benvenuti

Mon, 18 Sep 2006 14:35:56 +0000

Welcome to my blog! this is my first post…