Dashboard for evaluating the architecture of ICT projects in the Portuguese Public Administration

(1)

Dashboard for evaluating the architecture of ICT

projects in the Portuguese Public Administration

António Filipe Costa de Barros

Thesis to obtain the Master of Science Degree in

Engenharia Informática e de Computadores

Examination Committee

Chairperson: Prof. José Manuel Nunes Salvador Tribolet

Supervisor: Prof. André Ferreira Ferrão Couto e Vasconcelos

Co-Supervisor: Prof. Miguel Leitão Bignolas Mira da Silva

Member of the Committee: Prof. Pedro Manuel Moreira Vaz Antunes de Sousa

(2)

(3)

iii

Agradecimentos

ste trabalho foi caracterizado por muitos altos e baixos que me fizeram pensar, por

muitas vezes, se conseguiria chegar ao fim. Foi devido a algumas pessoas, que nunca

me deixaram desistir e deram sempre força, que consegui chegar ao objectivo que

queria, e neste momento estou a escrever este trabalho.

Em primeiro lugar gostaria de agradecer ao meu Orientador Prof. André Vasconcelos,

por todo o apoio, persistência e paciência demonstrada ao longo do trabalho.

Também gostaria de agradecer ao meu Co-Orientador Miguel Mira da Silva, por todas

as sugestões, disponibilidade, acompanhamento e criticas ao meu trabalho.

À AMA – Agência de Modernização Administrativa, e em especial à Dr.ª Maria João

Marques por todo o apoio e interesse demonstrado no meu trabalho durante a

realização deste.

Aos meus amigos e colegas de curso, um grande obrigado por estarem nos momentos

bons e menos bons, onde só me davam palavras de apoio e força para que acabasse

este trabalho.

Por último, mas não tão menos importantes, um especial agradecimento e dedicatória

à minha família, especialmente à minha mãe, pai e irmãos que sempre me acreditaram

em mim e transmitiam força nos momentos mais difíceis, quer na realização deste

trabalho, quer no decorrer do curso. Também um especial agradecimento à minha

namorada por toda a paciência que teve comigo, ao longo do curso e em especial

neste trabalho. Um muito obrigado, também por toda a motivação que ela transmitiu

ao longo destes 5 anos.

Sem estas pessoas não conseguiria ter chegado ao fim desta etapa.

(4)

(5)

v

Abstract

n the Portuguese Public Administration, public agencies need to propose projects to be allowed to spend money. Since there are limited financial resources it is necessary to restrict the approved projects to those that comply with the ICT guidelines created by the Portuguese Public Administration. With this objective, it was approved a measure that involves the assessment of all ICT projects, however the architecture evaluation is performed in a non-formal way, without the use of metrics to assess whether the project is consistent with the guidelines ICT.

In this work we propose a process for assessment of the architecture using metrics, to evaluate if a project is complying with the ICT guidelines. First this process will validate if the Open Standards are being respected and then it will calculate architectures metrics to validate that the project complies with the guidelines imposed by the Public Administration.

To demonstrate this process it was created an artifact. It was used the architecture of the agency responsible for the evaluation of all projects submitted – Agência para a Modernização Administrativa (AMA). With this artifact it’s possible to compare the as-is situation with the

project architecture (to-be) and see if there are improvements on the architecture.

We expect that the project evaluation stops to be made in a manual and ad-hoc way, and going to be done in an automatic, more formal and efficient way, in which all projects are subject to the same evaluation conditions.

Keywords: ICT Projects; Open Standards; Architectural Metrics; Architecture Evaluation.

(6)

(7)

vii

Resumo

a Administração Pública Portuguesa, as várias entidades públicas precisam propor projectos para serem autorizadas a gastar dinheiro. Com recursos financeiros limitados é necessário restringir os projectos aprovados para aqueles que cumprem com as directrizes TIC criadas pela Administração Pública Portuguesa. Com esse objectivo, foi aprovada uma medida que envolve a avaliação de todos os projectos TIC, no entanto a avaliação de arquitecturas é realizada de uma forma pouco formal, sem recurso a métricas que avaliam se o projecto está de acordo com as guidelines TIC.

Neste trabalho é proposto um processo de avaliação de arquitecturas com recurso a métricas, para avaliar se o projecto está em conformidade com as directrizes TIC. Em primeiro lugar, o processo irá validar se as Normas Abertas estão a ser respeitadas, em seguida, irão ser calculadas métricas arquitecturais para validar que o projecto está em conformidade com várias directrizes impostas pela Administração Pública.

Para demonstrar este processo foi criado um artefacto. Foi utilizada uma arquitectura da entidade que é responsável pela avaliação de todos os projectos submetidos - Agência para a Modernização Administrativa (AMA). Com este artefacto é possível comparar a situação actual (as-is) com as alterações que o projecto propõe (to-be) e verificar se existirão melhorias na

arquitectura.

A avaliação dos projectos deixa de ser feita de uma forma manual e ad-hoc passando a ser

realizada de uma forma automática, formal e mais eficiente, já que todos os projectos estão sujeitos às mesmas condições de avaliação.

Palavras-chave: Projectos TIC; Normas Abertas; Métricas Arquitecturais; Avaliação Arquitectural.

(8)

(9)

ix

List of Tables

Table 1. NPOS metric (adapted from [21]) ... 12

Table 2. Business Interfaces Required Factor Metric (adapted from [22]) ... 15

Table 3. Open Standards categorization ... 19

(13)

(14)

xiv

List of Figures

Figure 1. Design Science Research Process [12] ... 3

Figure 2. Graphic representation of the relation between Enterprise Architecture (EA) and Information System Architecture (ISA) ... 7

Figure 3. “Formulário de parecer prévio” steps ... 8

Figure 4. UML Metamodel of CEO Framework [5] ... 11

Figure 5. ArchiMate extended metamodel [22] ... 14

Figure 6. Generic evaluation process ... 17

Figure 7. Technology Services before extension ... 18

Figure 8. Evaluation application flow ... 20

Figure 9. Dashboard login window ... 27

Figure 10. Open Standards window blank ... 28

Figure 11. Open Standards validation ... 29

Figure 12. Architecture metrics window blank ... 30

Figure 13. Architectural metrics computation ... 31

Figure 14. Applications Modularity Factor chart ... 32

Figure 15. Open Source in IT Systems Factor chart ... 33

Figure 16. External IT Systems Integration Factor chart ... 34

(15)

(16)

1

1. Introduction

here is a large misalignment between costs spent on a project, and the benefits that derive from it. In order to balance this relationship, it was created a global plan for the rationalization of costs of IS/IT in Public Administration of Portugal (PGETIC – Plano Global Estratégico das TIC) [1].

This plan includes 25 measures of rationalization divided in 5 main categories:

• Improving governance mechanisms; • Cost reduction;

• Implementation of ICT solutions in common;

• Using IS/IT to enhance administrative change and modernization; • Stimulating economic growth.

The second category is one of the more important because it allows a better management of the money and that is the principal objective of PGETIC. The first measure of this category is the ICT Project and Costs Evaluation. Therefore, it must be created an evaluation process mandatory for all Information and Communications Technology (ICT) projects and can only be funded projects that ensure a real contribution to the Public Administration.

Although it is not possible to accurately estimate the savings resulting from the implementation of a mandatory preliminary assessment, it is estimated real savings within 5 years of 280 million euros [2].

This work will focus on the evaluation of the architecture of projects that are submitted for evaluation. Since there is no tool to evaluate and determine which projects should be funded or not, there is the problem of the Public Administration to be spending money on projects that in the beginning are unable to provide benefits to the Government. Only projects that are proven that will meet the ICT guidelines should be financed [3].

A central point of these guidelines is to validate the use of Open Standards. This is not only a concern of the Portuguese Public Administration, but it’s a concern in many Public Administrations, such as the UK [4].

The solution for this problem it’s a mechanism to evaluate if the system architecture of the project has quality enough for be approved. This quality will be measured through a set of metrics. A metric can be defined as “a quantitatively interpretation of the observable architectural attributes” [5].

The metrics will be calculated using a created application, which will be powered by XML files which reflect the architecture that is being assessed. The result of this application is a series of charts in which one the user can see the result of the various metrics through different vectors (strategic alignment, costs, risk).

(17)

2

1.1. Research Problem

The current assessment process doesn’t make an evaluation of the architecture of projects in an automatic and formal way.

As studied in other works, most projects of Information Systems (IS) fail. In 2003, in UK, only 16% were successful and 41% of projects were over budget or didn’t meet the deadlines [6].

There are several investment appraisal techniques: Economic Ratio, Economic Discounting, Strategic, Integrated, Analytic Portfolio and Other Analytic Analysis [7].

Economic Ratio Analysis is the most widely used for project evaluation, but this appraisal techniques do not work in IS/IT environment [8]. These appraisal techniques are based on “… cash values, to tangible benefits and costs but largely ignore project, or event risk, non-financial and intangible IT/IS implications” [7]. There are already some frameworks that help to create representations of

information systems architecture [9] [10]. Nevertheless, the automatic evaluation of these architectures (i.e. comparing two architectures and know which is best in a particular vector, for example, the risk) is still not well developed.

The plan of PGETIC is already in course, using an Excel document to evaluate the costs, benefits, risk and strategic alignment. However the assessment of the project architecture is being made in a manual way, without the use of metrics calculated automatically. Moreover, the little study of metrics for evaluation an architecture is a critical problem in Enterprise Architecture as already discussed by other authors [11]. Without an automatic assessment of the project architecture submitted, the project evaluation will be overdue and performed in an adhoc way, because there is no formal assessment applied to all the projects.

1.2. Research Methodology

With this work is expected to perform a work with rigor and scientific validity. For this it is necessary to use a research methodology.

(18)

3

This research methodology has six activities, like is shown in Figure 1.

Figure 1. Design Science Research Process [12]

Below is explained briefly each activity. More detail can be found in [12].

Problem identification and motivation – In this activity it’s necessary to define the specific research problem and justify the value of a solution. Justifying the value of a solution has two objectives: to motivate the researcher and the audience of the research and it helps to understand the reasoning associated with the researcher’s understanding of the problem. Knowledge of the state of the problem and the importance of its solution are the resources required.

Define the objectives for a solution – Infer the objectives of a solution from the problem definition and knowledge of what is possible and feasible. These objectives should be inferred from the problem specification. Resources required are the state of the problems and current solutions, if any, and their efficacy.

Design and development – In this activity the solution artifact is created. This activity includes determining the artifact’s desired functionality and its architecture and then creating the actual artifact. Resources required are knowledge of theory that can be brought to bear in a solution.

Demonstration – Demonstration of the use of the artifact to solve one or more instances of the problem. Resources required are the effective knowledge of how to use the artifact to solve the problem.

Evaluation – In this activity it’s observed and measured how well the artifact supports a solution to the problem. This activity involves comparing the objectives of the solution with the results obtained with use of the artifact in the demonstration.

(19)

4

This methodology steps are mapped with the following chapters of this document: Problem identification and motivation is presented in Chapter 1 – Introduction. Define the objectives for a solution is in Chapter 1 – Introduction and Chapter 2 – Related Work. The Design and development

phase can be found on Chapter 3 – Proposal. The Demonstration phase and Evaluation can be found in Chapters 4 and 5 respectively.

1.3. Structure of the document

In this document, starting with Chapter 1, it’s presented a motivation for this work, the problem that it’s addressed and the research methodology followed.

In Chapter 2 it’s presented a review of some work related with this problem and explained why these works doesn’t solve the problem in question. Some of these works will be used in the proposal so it is important for the reader to know some of these aspects that will be discussed in the solution description.

Following to Chapter 3 it’s describes the proposal that this work make to solve the problem that was described. Thereafter, in Chapter 4 it’s presented the artifact created and how the solution proposed was put in practice.

(20)

(21)

6

2. Related Work

his section of this work is intended to be a review of related knowledge and work that might be relevant in this dissertation. First are clarified three important concepts for the understanding of this report, the concept of architecture of information systems, ArchiMate and the description of one of the measures of PGETIC, i.e., the current process of project evaluation. Then it’s described some studies about the process of enterprise architecture evaluation and studies that have attempted to create metrics for evaluating architectures – first it’s presented metrics created based in a framework known as CEO Framework, and next it’s presented metrics created from architectural principles, based in Archimate.

2.1. Architecture of Information Systems

The IEEE Standard 1471-2000 defines architecture as “the fundamental organization of a system incorporating its components, and the relationships between them with the environment and the principles guiding its design and evolution” [13].

In the context of Information Systems is usual to divide the concept of architecture in three different perspectives: Software, Enterprise and Information System.

The main study of Software Architecture (SA) is how programs or applications components are internally built [14]. At this level it is important to consider the objects and classes needed for implementing the software.

Enterprise Architecture (EA) is “a coherent whole of principles, methods, and models that are used in the design and realization of an enterprise’s organizational structure, business process, information systems and infrastructure” [15].

Information System Architecture (ISA) is the representation of information systems, their relations, principles and guidelines for the purpose of supporting the business [16].

(22)

7

ISA is usually divided into three sub architectures [17]:

•

Information Architecture – represents the main data types that support business;

•

Application Architecture – defines applications needed for the business support;

•

Technological Architecture – represents the main technologies used by the applications.

The EA is considered a vaster concept than ISA because it focuses on the company as a whole (including Business Architecture and Organizational Architecture), while the ASI focuses on systems that move, change and store information. The relationship between the AE and ASI can be defined based on Figure 2.

Figure 2. Graphic representation of the relation between Enterprise Architecture (EA) and Information System Architecture (ISA)

2.2. Archimate

ArchiMate (technical standard from The Open Group1_{) is a modeling language that allows architects to}

make a detailed description of the enterprise architecture. Enterprise architecture is developed to detail the structure and the relationship of the enterprise.

Using an architecture framework will speed up and simplify the architecture development and communication with the stakeholders, ensuring complete understanding of the designed solution.

An architecture framework is used to structure the concepts and relationships of the ArchiMate language. It divides the enterprise architecture in to a business, application and technology layer.

(23)

8

•

Business Layer: offers products and services to external customers which are realized by

business process performed by business actors;

•

Application Layer: supports the business layer with application services which are realized

by software applications;

•

Technology Layer: supports the application layer and offers infrastructure services need to

run applications.

2.3. Project and Costs Evaluation

After the 15th of September 2012, all projects of public administration require approval before being initialized. The organization responsible for evaluating projects is the AMA (Agência para a Modernização Administrativa) .

The support tool that is used to evaluate the projects consists in an Excel sheet (called “Formulário de parecer prévio”) and in a To-Be scenario (called “Novo Cenário Arquitectural (NCA)”). These two

documents will be sent to AMA to be evaluated and will result in the approved or rejected project.

All information regarding this process can be seen on the website of this PGETIC measure2.

The Excel sheet is divided in five steps, as shown in Figure 3.

Figure 3. “Formulário de parecer prévio” steps

In this work it will focus on the vector of Cost, Strategic Alignment and Risk. This are the information that can be viewed and evaluated through the project architecture, and that’s the focus of this work.

(24)

9

2.4. Enterprise Architecture Evaluation

2.4.1. Introduction

The evaluation of enterprise architectures is not a well studied area, although there are already some works that try to address this problem. In this section it will be explained some of this studies and why these does not solve the problem that this paper is addressing. Some of these works are focused on the enterprise architectures analysis and not on the practice of evaluating a specific architecture. Nevertheless these studies were analyzed to understand the approach to perform the analysis of architectures.

2.4.2. Architecture Evaluation Studies

In work [18] the authors propose a tool for enterprise architecture analysis based on models. This method has three phases: Assessment Scoping, Evidence Collection and Analysis. The first one results in a series of potential future scenarios. In the second one it’s necessary to do a collection of information that will be used in the architecture evaluation. In the last phase the quality attributes are calculated and the models are presented in the form of enterprise architecture diagrams.

Like is shown on work [19] the analysis of an enterprise architecture have 4 phases. The first phase is concerned with the formulation of different scenarios to be evaluated. It follows the definition of evaluation criteria that will be used. Then makes the analysis of the architecture and finally it is made the choice of which scenario should be approved. In this work the authors argue that a good architectural theory it’s necessary to make good models leading to good decisions.

(25)

10

2.4.3. Critical Analysis

The first work descripted do not solve the problem of this paper because the second phase of the method is very time consuming because it’s needed the collection of information in the form of questionnaires and reading documents and that is impossible with the amount of received projects to evaluate.

The second study does not solve the problem that is addressed in this paper because the scenario formulation is not the responsibility of the agency that will evaluate the projects and could happen that some projects could not be evaluated with the theories that were defined.

Related to the third and last reviewed work, in the context of this thesis, the first stage of this analysis is the formulation of the architecture of the projects carried out by various public entities. Next, it should be defined the evaluation criteria. These criteria will be the most important vectors for the Public Administration: strategic alignment, costs and risks. In relation to the last phase of the analysis, as all public entities are different, it would be necessary to have different weighted criteria for each of them, as, for example, the criteria of costs may not be the most important for all public agencies.

2.5. CEO Framework and Metrics

2.5.1. Introduction

CEO Framework provides a formal way of describing business goals, process and resources. This framework is composed of three separate levels.

The first level describes the current set of goals that drive business. The second layer – business processes – gives support for the first layer. This second layer must exist to satisfy the goals. Business processes interacts with resources and may be supported by information systems (third layer). The information system layer is important because has the aim to model the components of the system that support business [21].

(26)

11

2.5.2. Metamodel

Figure 4 presents the UML Metamodel defined for the CEO Framework.

Figure 4. UML Metamodel of CEO Framework [5]

To model Information System Architecture (ISA) key concepts the Block component was specialized. The concepts for the ISA are:

• Information Entity – person, place that is relevant in the business context;

• IS Block – mechanisms and operations organized in order to manipulate organization data; • IT Block – the infrastructure that implements an IS Block;

(27)

12

2.5.3. ISA Evaluation Metrics

In the study [21], the authors proposed a series of metrics based on the research of authors who specialize in other areas (security, scalability). These metrics were adapted to the ISA, and uses the CEO Framework explained above. The authors argue that with these metrics, architects can measure the impact of each of their decisions, and thereby are better equipped to build an architecture aligned with the desired qualities.

Among the set of metrics created it is presented one below.

Acronym NPOS

Name Average Number of Possible Operation Systems

Computation

= #

– the number of possible operation systems families that the IT Application Block supports.

#IT Application Block – the number of IT Application Block in the ISA.

Architectural

Levels Technological Architecture ISAQualities

The Portability and Technical Interoperability of an Enterprise Information System tend to increase with this metric

Table 1. NPOS metric (adapted from [21])

With this metric when comparing two architectures is possible to check which one is more portable, i.e. which can run on more operating systems.

2.5.4. Critical Analysis

Like is said in [5], these metrics has to be better studied, because they are still immature. Further research is necessary to clarify the metrics impact on the quality attributes.

If the metrics were applied in Archimate would have more impact because this framework is currently he standard used for modeling enterprise architectures.

(28)

13

2.6. Architectural Principles and Metrics

2.6.1. Introduction

A series of metrics to measure the quality of an architecture from architectural principles are proposed by Vieira [22].

A principle can be used in the evaluation process, since it is considered an optimal solution. “Architecture principles fills the gap between high-level strategic intentions and concrete design decisions” [23] – ensure that Enterprise Architecture is future directed.

In [23] are proposed 59 principles, which were gathered from actual cases. These principles were classified in two dimensions: the quality attributes and the architecture level impacted (business, data, application, technology).

(29)

14

2.6.2. Metamodel

In this work, the authors made an extension to the Archimate metamodel. The UML was chosen to represent the Archimate model, like is shown in Figure 5.

Figure 5. ArchiMate extended metamodel [22]

(30)

15

2.6.3. Evaluation Metrics

Using the architectural elements identified and the extended metamodel, the authors defined a “formula” that allows to measure the suitability of a certain architecture decision facing a principle. One example of these metrics is presented in Table 2.

Name Business Interfaces Required Factor

Computation

! 2 = # #$ $$ !%

&

#Business Service = the number of Business Service on the Architecture

BSIR = is the value of the attribute “interfacesRequired” for the different Business Service on the Architecture.

Quality(ies) Usability and Efficiency increase with this metric.

Rationale

We want to measure the number of interfaces required to execute a service. According to principle A.2, having a single point ensures that consistent information is provided to the customer; Is more efficient to dedicate resources to handling customer contacts.

Table 2. Business Interfaces Required Factor Metric (adapted from [22])

2.6.4. Critical Analysis

In this work, the author has endeavored to apply architectural principles, to a modeling tool, which is currently the standard, the ArchiMate.

The ArchiMate metamodel had to be extended because some concepts that were important to the conception of the metrics, did not exist. Also the inclusion of attributes for each of the elements is an extension to ArchiMate metamodel.

(31)

(32)

17

3. Proposal

he solution for the problem presented involves an evaluation process for assessing the project architecture submitted, as shown in Figure 6. This process was based on another process with the aim of creating an ISA (see Appendix D). In this work we are not interested in creating an ISA (scenario as-is and to-be) as this is the responsibility of public agencies. What this work proposes can be seen as a sub-activity of the process "Avalia ASI" which Vasconcelos proposes in his work [16]. To

implement this process it was necessary to make several changes to the metamodel in use, and decisions had to be taken about choosing which metrics and criteria to measure. This will be described and explained throughout this Section.

Figure 6. Generic evaluation process

The first concern in the evaluation process is the assessment if a certain project complies with the Open Standards imposed by the Portuguese Public Administration [24]. A list all of this Open Standards can be found on Appendix A. This type of verification was already done in Excel sheet explained in Section 2.1.3, however this validation was not detailed - it was not possible to know what Open Standards were not being respected. More, this is the type of information that should be incorporated into the architecture of the project submitted.

There are two types of Open Standards - the obligatory and the recommended. If an obligatory Open Standard is violated, the project evaluation should be immediately stopped, and the project should not be approved. Projects with a large number of recommended standards in use should be benefited.

After this validation is necessary to validate if the project architecture being evaluated complies with the guidelines ICT reference imposed by the Public Administration. For this, it is necessary to

(33)

18

implement architectural metrics that will measure this compliance. To know which metrics should be implemented it was necessary to define what criteria should be measured. These criteria should be the areas that the Portuguese Public Administration considers to be critical in the evaluation of an ICT architecture. So were chosen the following areas: strategic alignment, costs and risk. With this categorization is possible to verify in which area the project suggests improvements.

The metrics that should be implemented must be aligned with the three categories which were described above. For this, it was used the metrics already created and described in the related work, and were selected the ones that fit best in these areas. It were only considered metrics that were created from the architectural principles, since they were created using the framework Archimate, which is the basis of the metamodel used in this work. It was analyzed all metrics and based on the description of each and were selected to fit best in each of the areas described above.

Thus, it is possible to check whether a particular project will lead to current architecture towards the ICT guidelines. This is possible by comparing the result of a given metric in as-is, with the changes to the architecture proposed in the project (to-be).

The Portuguese Public Administration was in the process of redesigning the current metamodel. In this study we used the metamodel that was being created that would come into use soon (see Appendix B). This metamodel is inspired in the modeling language ArchiMate (Section 2.2).

To be able to assess the ICT reference guidelines for Public Administration, it was necessary to extend the metamodel that was being created. This extension involves the incorporation of the Open Standards in the Technology Services and by adding some attributes of several entities.

The Technology Services were not detailed and there wasn’t any information about the different Open Standards in the new metamodel, like is shown on Figure 7.

(34)

19

To incorporate the relevant Open Standards [24] in the metamodel and thus can make an assessment of what standards would be respected, it was used the Technical Reference Model (TRM) and the Standard Information Base (SIB) of TOGAF [25]. The TRM provides the division and structure of the various Technology Services existent (see Figure 7), while the SIB provides a database of standards to define the particular services.

Some of the Open Standards were already categorized on SIB, but it was necessary to extended with the standards that weren’t categorized. To categorize the Open Standards that were not present in the SIB, it was used the description of the various Technology Services in TRM. As we can see in Appendix C, the metamodel is extended with the various Open Standards classified in each corresponding service.

The list of standards and service where it was categorized is shown in Table 1.

Technology Services

Open Standard

Classification

Data Interchange Services

XML Mandatory

XSLT 2.0 Mandatory

XSD Mandatory

XSL 1.1 Mandatory

ODF 1.1 Mandatory

ATOM 1.0 Mandatory

WCS Mandatory

WFS Mandatory

WMS Mandatory

WPS Mandatory

Data Management Services SOAP 1.1 Mandatory

Graphics and Imaging Services PNG Recommended

Network Services

CalDav Mandatory

HTTPS Mandatory

WebDav Recommended

RTSP Mandatory

IMAPS Recommended

POP3S Recommended

SMTPS Recommended

Security Services

TLS 1.0 Mandatory

SAML 2.0 Mandatory

WS-RM 1.1 Recommended

WS-Security 1.2 Recommended WS-Security Username Token

Profile 1.0 Recommended

Software Engineering Services Javascript 1.5 Recommended User Interface Services WCAG Mandatory/Recommended

Table 3. Open Standards categorization

(35)

20

The artifact has as input XML files which reflect the architecture that is being evaluated. This format was chosen because it is a standard of data exchange. With this information it’s possible to evaluate the Open Standards and calculate the architectural metrics. This application also has the export function of the values of the metrics to an XML file. With this file it’s possible to incorporate these metrics in the evaluation process already in action. The application flow is shown in Figure 8.

Figure 8. Evaluation application flow

Below, it will be detailed the changes that were necessary for the creation of the artifact, in particular in each of the two main phases – Open Standards Verification and Architectural Metrics Computation.

3.1. Open Standards Verification

The first phase of the evaluation process is to verify if any Open Standards are violated. For this it was necessary to add two boolean attributes in every Technology Services:

• Relevant for this project? – This attribute has the objective to know which Open Standards are relevant for that specific project.

(36)

21

In case there is a mandatory standard that is relevant to the project, but will not be used, the project evaluation is stopped automatically. However, if the standard is recommended, the evaluation continues to the next stage, but is presented which standards are being violated.

To validate some more specific Open Standards was necessary to add new attributes which will be explained below.

There are services that implement an additional security layer to that already existed this is the case of HTTP / HTTPS for example. To be able to distinguish if it’s needed this additional security layer was added a new attribute. This boolean attribute is called Secure access/sending?. In the case of HTTPS, if a project has this attribute at True, but the attribute Use it in this project? at False, the project is automatically rejected because the standard HTTPS is required when there is need for extra security.

The WCAG 2.0 standard validates the level of accessibility according to the information that will be available. For that were created three attributes: Offers exclusively information and contents?,

Offers online services? and Accessibility Level (A / AA / AAA). The first two are boolean attributes, while the third is a text entry where is completed with the level of accessibility. The artifact validates the information that is entered and checks if the level of accessibility is consistent with what was stipulated in the Open Standards document. For example, for a project that has Offers online services? at True, it’s necessary to have the Accessibility Level (A / AA / AAA) with the value AA to be approved.

The complete list of attributes added to validate the Open Standards is on Table 4.

Attribute

Technology Service

[Boolean] Relevant for this project? All [Boolean] Use it in this project? All

[Boolean] Secure access/sending?

HTTP HTTPS IMAP 4 IMAPS POP3 POP3S SMTP SMTPS [Boolean] Offers exclusively information and

contents? WCAG 2.0

(37)

22

3.2. Architecture Metrics Evaluation

The three most important categories for Public Administration are the costs, risks and strategic alignment. The metrics chosen to be implemented are those which are best suited in these three categories. To make this categorization it was used each of the metric description.

The metrics chosen to incorporate the application are the following:

• Applications Modularity Factor; • Open Source in IT Systems Factor; • External IT Systems Integration Factor; • Business Interfaces Required Factor; • Recommended Open Standards Factor; • Citizen Card Authentication Factor.

The first 4 metrics where based on work of other authors [22] and the other two metrics have been developed for assessing whether the project is consistent with the most important ICT guidelines [3].

In the following sections it will be described the changes that were necessary for computing the metrics.

3.2.1. Applications Modularity Factor

This metric is measuring the modularity of applications, that is, if a particular application depends on another. The ideal case is that each application is independent of the other, for in the event of a crash, not requiring the shutdown of other applications. For this computation it was necessary to add a boolean attribute in the applications entities to know if an application depends on another. With this information the artifact subtracts one from the division of the number of applications with the attribute

(38)

23

3.2.2. Open Source in IT Systems Factor

To calculate this metric is necessary to have information on what software is Open Source or not. For this, it was created a boolean attribute in the Software entities displaying this information. The artifact counts the number of entities with this attribute at True and divides with the sum of all Software entities. The ideal outcome occurs when all software that exist in the organization have the attribute Is Open Source? at True. Like is said in Appendix E.b, with this type of software the costs of software acquisition are lower making sense of this metric to be incorporated into the vector of costs.

3.2.3. External IT Systems Integration Factor

In this metric it’s measured if an application connects with other external IT within an organization. The authors argue that it’s more efficient and cheaper if the interfaces costs are spent only once and if it’s necessary to change anything it’s limited to one component. To calculate this metric the artifact computes 1 divided by the sum of applications with the new boolean attribute Connects to External IT? at True. The value of this metric is 1 when exists only one application that connects to external IT, which is the ideal outcome. In Appendix E.c it’s said that using a dedicated IT component to integrate with external IT, the costs are spent only once. Thus, this metric makes sense to be incorporated into the vector of costs.

3.2.4. Business Interfaces Required Factor

(39)

24

3.2.5. Recommended Open Standards Factor

This metric has the objective to benefit projects with higher number of recommended Open Standards being used. This metric counts the number of recommended Open Standards that are relevant (#RelevantOS) and of those which will be used in the project (#UsedOS). The formula is the following:

RelevantOS

#

UsedOS

#

=

Metric

d

Recommende

The optimal value is 1 when all the relevant Open Standard for that project will be implemented.

This metric was incorporated in the vector of strategic alignment because it’s one of the major ICT guidelines for the Portuguese Public Administration.

3.2.6. Citizen Card Authentication Factor

Another metric were created to evaluate whether the guideline regarding the authentication by the Citizen Card is respected. This metric calculates the ratio between the number of application that needs authentication and are using the Citizen Card authentication (#CitizenCardApp) and the total number of applications that needs authentication (#TotalApp). The formula is the following:

TotalApp

#

dApp

CitizenCar

#

=

Metric

tion

Authentica

One more time, the optimal value of this metric is 1. This happens when all applications that require authentication are made through the Citizen Card.

As the metric described above, this metric has been incorporated in the vector of strategic alignment, because it is also one of the ICT guidelines that must be respected.

(3.1)

(40)

(41)

26

4. Demonstration

n this section it will be explained how it was performed the demonstration of the proposal of this work. In the first section is described which software is used for modeling the architecture, next it’s described the artifact created and then the artifact is applied to a specific architecture.

4.1. Enterprise Architecture Management System

(EAMS)

In this demonstration it was used a software program that manages and maintains the enterprise architecture updated – Enterprise Architecture Management System (EAMS). This software allows gather information from various data sources such as Microsoft Office documents (Excel, PowerPoint, Word and Project) or tools and systems (Web Services, Visio and other specific systems) [26]. This information is used to automatically generate blueprints, thereby reducing effort and costs. In EAMS it’s possible to have multiple attributes for an entity, allowing storing more information about an entity.

This software it’s been used in the Public Administration to visualize and maintain the different project architectures submitted. As the new metamodel was not implemented in EAMS, it was necessary to model the architecture we were evaluating, using a programming language that the EAMS offers. This language, called EALang, is used to populate a database, where the EAMS will read and create the blueprint to be able to view the architecture.

This software was used to modeling the architecture that was been evaluated and then was exported for XML format.

(42)

27

4.2. Dashboard for architecture evaluation

The first window that appears after starting the dashboard is the Login window, which was implemented for security reasons. Within the evaluation team of architectures may be several people with different roles. With this measure the dashboard is restricted to those who have the credentials (see Figure 9).

Figure 9. Dashboard login window

After the insertion of credentials the dashboard presents a window where it’s selected the location where the architecture formatted in XML files are stored as well as the evaluation results of the Open Standards.

(43)

28

Figure 10. Open Standards window blank

(44)

29

Figure 11. Open Standards validation

(45)

30

Figure 12. Architecture metrics window blank

(46)

31

Figure 13. Architectural metrics computation

4.3. Artifact Demonstration

In the context of this research demonstration, it was used the architecture of a public agency –

Agência para a Modernização Administrativa (AMA). As previously mentioned, this agency is the

responsible to make the assessment of the projects of all the entities of the Portuguese Public Administration. Although this was not the factor of choice for the selection of this organism, it is interesting to notice if the architecture of the reviewer complies with the standards that are evaluated in all projects.

It is also important to notice that this architecture has some months and the ICT guidelines were defined after the collection of the architecture. Although the architecture does not have all the information to calculate all the metrics implemented, serves to demonstrate the use of the tool for the project evaluation and the current status (as-is).

(47)

32

Among other information, it was also necessary to collect information about which software is Open Source since the architecture did not contain this kind of information.

After exporting the data via the export function of EAMS, we'll have all the architecture that we are evaluating, in XML format. With this it is possible to evaluate the architecture, using the tool created. As we are dealing with a scenario as-is, before the implementation of Open Standards it will not be possible to demonstrate this evaluation.

Moving on to the second phase of evaluation metrics of architecture it was possible to calculate the following metrics:

A. Applications Modularity Factor; B. Open Source in IT Systems Factor; C. External IT Systems Integration Factor; D. Citizen Card Authentication.

For the first metric, the resulting chart can be found in Figure 14. There we can see that there are 15 applications with dependencies against 8 applications without dependencies. With this graphical view of the metric, it is possible to verify in a fast way that this architecture has a high risk as there are several applications depending on another which is not an ideal situation, according to the metric Applications Modularity Factor.

With this artifact, it is also possible for the evaluator to know what applications are those that depend on other, by simply press on the corresponding area of the chart. This feature applies to all charts generated by the application.

(48)

33

For the metric corresponding to the Open Source the chart is in Figure 15. To compute this metric was necessary to collect information about the software used in the organization. The architecture that was being used did not contain the information which Software is Open Source or not. This information can be seen on Appendix F.

Once again, we can see in the chart, that in this architecture, the not Open Source Software is the majority. Thus, it is possible to confirm that the guideline corresponding to the use of Open Source Software is not being respected in the organization.

Figure 15. Open Source in IT Systems Factor chart

For the External IT Systems Integration Factor metric the results can be viewed in Figure 16. With an optimal value of 1 application that connects with external IT, we can see that in this architecture are 5 applications that connect with external organisms. With a higher number of these applications the interface costs are spent more times and if it’s necessary to change anything it will be necessary to modify in 5 components instead of 1.

It was necessary to do a collection of applications which communicate to the outside of the organization. In the Portuguese Public Administration there is a component that is responsible for include all interfaces that are necessary for communication outside of the various entities – Plataforma de Interoperabilidade. This component is transversal to the Portuguese Public Administration and

consists of other components: Plataforma de Integração, Fornecedor de Autenticação, Plataforma de Pagamentos and Gateway SMS. All these components were modeled in this architecture, but between

these 5 components only the Plataforma de Interoperabilidade was considered that communicates

(49)

34

Figure 16. External IT Systems Integration Factor chart

Regarding the authentication by the citizen card, we can see by the chart in Figure 17 that most applications have authentication using the citizen card. Thus it can be seen that the guideline corresponding to the use of authentication by the Citizen Card is respected in this organization.

(50)

35

The outcome of the export of calculated metrics, gives the following results:

A. Applications Modularity Factor = 0.35 B. Open Source in IT Systems Factor = 0.21 C. External IT Systems Integration Factor = 0.2 D. Citizen Card Authentication = 0.91

(51)

(52)

37

5. Evaluation

n this section it will be explained how the assessment of the artifact created was conducted. Initially it’s presented the evaluation supported by the principles of Österle and then it’s presented a discussion of the results shown.

5.1. Österle Principles

To evaluate the artifact created we used the Österle principles [27]. This work considers the artifacts created must comply with the following principles: abstraction, originality, justification and benefit. Next it is shown that the artifact created is complying with the 4 principles.

• Abstraction – The artifact is applicable to all architecture projects submitted and to as-is architectures using the new metamodel.

• Originality – Although some metrics that were used to evaluate the architecture have already been created by other authors, the automatic tool did not existed, as well as the metrics for measuring the compliance of ICT guidelines.

• Justification – The artifact is justified since it uses already published papers in conferences. • Benefit – With the artifact created the evaluator agency use a formal and automatically way to

evaluate the projects submitted leading to a shorter evaluation time.

5.2. Results Discussion

With the demonstration in the architecture of AMA, it was simple to see that the evaluation, using the artifact created, is more intuitive and formal that was before. With this dashboard is possible to verify in a quick way, which guidelines ICT is being the most respected in the public agency. In this case, among the metrics that were calculated, it is clear that the guideline that is being more respected is

(53)

38

related to the authentication using the citizen card. Thus, it is possible to gather attentions and efforts for the other ICT guidelines to be increasingly respected, namely, that the corresponding metric approaches to 1.

Also there is already an effort to make the metric External IT Systems Integration Factor to be improved, and the proof is the existence of the Plataforma de Interoperabilidade previously explained.

With this dashboard the evaluation became independent of the person who is doing because with this artifact all the projects submitted will be evaluated using the same metrics, in the same conditions. Because of this it will be possible to compare the as-is architecture with the changes that the project recommends and have an idea of what vector can be improved if the project is approved, based on the results given by the artifact.

Also, with this dashboard can also be seen which vectors, a certain public agency may improve, and do not approve projects which do not propose improvements in these vectors. For example, as shown in the demonstration, the metrics associated with the vector of costs have a lower result and may indicate that the public agency is spending too much money. With this information, in the future, it can be made a prioritization for only be approved projects that improve the vector of costs.

With the artifact created the proposal is proven to be implementable through a software application. With this, the evaluator agency can use this artifact to accelerate the architecture evaluation and reducing waiting time and queues of projects for evaluation. With this software application the evaluation of each of the different projects architectures may even be compared, since the various architectures are evaluated in the same conditions using a formal process.

(54)

(55)

40

6. Conclusion

n this chapter it will be presented the conclusions of the work conducted, the limitations and some future work that can be made to improve the actual proposal.

To balance the costs spent on a project, and the benefits to be derived therefrom, a national plan for the rationalization of costs of IS/IT in Public Administration of Portugal (PGETIC – Plano Global Estratégico das TIC). This plan has several measures divided by 5 categories. One of these measures

relates to the prior evaluation of projects proposed by the Portuguese Public entities. These projects should only be approved if the Open Standards are being respected and these projects are aligned with the ICT guidelines for the Public Administration. Although this measure is already being carried out, the assessment of the architecture of the proposed projects was carried out in an ad-hoc way.

To solve this problem, it was performed reviews of the related work, in particular, the enterprise architecture evaluation and evaluation metrics. Thus was created an evaluation process, which is evaluated first, if Open Standards are being met and then, some architectural metrics are computed. With this in mind, it was created an automatic tool capable of evaluating the architecture project. It was also necessary to allow this tool to assess the current state of the architecture, so that it can be verified the impact of the approval of a particular project, in the current status.

For the demonstration of this artifact it was used an architecture of a public agency, in this case the AMA. In this demonstration it can be seen the implementation of the artifact in a real case, as well as its value. With this demonstration it was proved that was possible to implement the proposal from an automatic assessment tool. Although it was not possible to compute all the metrics that have been implemented in the artifact, 4 of the 6 metrics generated results.

After this demonstration, the proposal of this work was evaluated with Österle principles and held a discussion of the results generated. It was proved that the generic process proposed in this work could be implemented with a software tool, although not all metrics have been possible to compute. Now the Portuguese Public Administration can use the dashboard created for the evaluation of architectures of ICT projects to confirm whether a particular project is going according to the ICT guidelines imposed. This analysis can be performed by comparing the outcome of a particular vector (costs, for example) of the current state with what the project proposes.

It is important to note that although this work is focused on evaluating projects of Portuguese Public Administration, this is easily applicable to other public authorities or even companies. For example, the application of Open Standards is already a hot topic in many companies and governments, so the metrics for this subject could be applied to various institutions. The implementation of new metrics and new Open Standards can be done in a quick and easy way using the artifact created.

(56)

41

The main results of this work can be divided into Primary contributions and Secondary contributions, which will be explained below:

• Primary contributions: with this work it was tempted to create an automatic process to evaluate architectures using a software tool. Another contribution of this work was the incorporation of some issues that are extremely important to the Portuguese Public Administration, such as the Open Standards in metamodel that will be used.

• Secondary contributions: to demonstrate this work was necessary to collect some information about the architecture that was being evaluated. This information that was collected can be important can be very important for those who develop work related to Enterprise Architecture in the same organization. Moreover, with this information the evaluator agency knows what attributes should be required for the associated metrics can be calculated. Another contribution was the incorporation of more standards in the Standard Information Base (SIB) of TOGAF.

6.1. Limitations of this work

Relating to this work, this can be improved. Then we will present some limitations that this work has:

• The dashboard is configured to receive XML files with a certain format, and this should be respected, as well as the names of these files;

• The dashboard gives no information whether a project should be evaluated or not, because it was necessary an integration between the data architecture and the Excel file and this was not the scope of this work;

(57)

42

6.2. Future Work

It is important to remember that the dashboard created is a prototype and this can and should be improved. Below is presented some suggestions for future work to improve the evaluation of architectures, and in particular the dashboard created.

• The dashboard can be integrated in an Enterprise Architecture Management software, like EAMS, eliminating the need for the user to export the architecture for XML files. With this feature the evaluation time will reduce;

• Implement new connectors for the dashboard to be able to read information from other sources, such as Web-Services;

• Implement and create new metrics in the dashboard;

• Implement the automatic comparison between the current architecture and to-be scenario; • Use an automatic evaluation method, to combine the value of the calculated metrics, with the

(58)

(59)

44

7. References

1 Diário da República. Decreto-Lei n. 107/2012. 2012.

2 Governo de Portugal. Plano global estratégico de racionalização e redução de custos nas TIC, na Administração Pública. 2011.

3 Agência para a Modernização Administrativa. Diretrizes TIC. 2012.

4 Cabinet Office. Open Standards Principles: For software interoperability, data and document formats in government IT specifications. 2013.

5 Vasconcelos, André, Sousa, Pedro, and Tribolet, José. Information System Architecture Evaluation: From Software to Enterprise Level Approaches. In 12th European Conference On Information Technology Evaluation ( 2005), 473-488.

6 Sauer, Chris and Cuthbertson, Christine. The State of IT Project Management in the UK (2003). 7 Irani, Z. and Love, P. E. D. Developing a frame of reference for ex-ante IT/IS investment evaluation

(2001).

8 Lin, Chad, Pervan, Graham, and McDermid, Donald. IS/IT Investment Evaluation and Benefits Realization Issues in Australia (2005).

9 Zachman, J. A. A framework for information systems architecture. IBM Systems Journal, 26, 3

(1987), 276 - 292.

10 Sowa, J. F. and Zachman, J. A. Extending and formalizing the framework for information systems architecture. IBM Systems Journal, 31, 3 (1992), 590 - 616.

11 Kaisler, Stephen H., Armour, Frank, and Valivullah, Michael. Enterprise Architecting: Critical Problems. Proceedings of the 38th Hawaii International Conference on System Sciences (2005),

224b.

12 Peffers, Ken, Tuunanen, Tuure, Rothenberger, Marcus A., and Chatterjee, Samir. A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems, 24, 3 (2008), 45-77.

13 Maier, M. W., Emery, D., and Hilliard, R. Software architecture: introducing IEEE Standard 1471 (2001), 107-109.

14 Garlan, David and Shaw, Mary. An Introduction to Software Architecture. Carnegie Mellon

University, 1994.

15 Lankhorst, Marc. Enterprise Architecture at Work: Modelling, Communication and Analysis.

Springer, 2009.

(60)

45

Instituto Superior Técnico, 2007.

17 Spewak, S. Enterprise Architecture Planning: Developing a Blueprint for Data, Applications, and Technology. Wiley, 1993.

18 Johnson, Pontus, Johansson, Erik , Sommestad, Teodor, and Ullberg , Johan. A Tool for

Enterprise Architecture Analysis. 11th IEEE International Enterprise Distributed Object Computing Conference (2007), 142 - 153.

19 Johnson, Pontus, Ekstedt, Mathias, Silva, Enrique, and Plazaola, Leonel. Using Enterprise Architecture for CIO Decision-Making: On the Importance of Theory. Proceedings of the Second Annual Conference on Systems Engineering Research (2004).

20 Simonsson, Mårten, Lindström, Åsa, Johnson, Pontus, Nordström, Lars, Grundbäck, John, and Wijnbladh, Olof. Scenario-based evaluation of Enterprise Architecture: A top-down approach for chief information officer decision making. 7th International Conference on Enterprise Information Systems, ICEIS 2005 (2005), 130 - 137.

21 Vasconcelos, André, Pedro, Sousa, and José, Tribolet. Information System Architecture Metrics: an Enterprise Engineering Evaluation Approach. The Electronic Journal Information Systems Evaluation, 10, 1 (2007), 91 -122.

22 Vieira, Tiago. Evaluating Enterprise Architectures: From Principles to Metrics. Instituto Superior

Técnico, 2012.

23 Greefhorst, Danny and Proper, Erik. Architectural Principles. Springer, Berlin Heidelberg, 2011.

24 Diário da República. Resolução do Conselho de Ministros n.º 91/2012. 2012.

25 The Open Group. TOGAF 9.1. 2011.

26 Sousa, Pedro, Gabriel, Ricardo, Tadão, Gustavo, Carvalho, Rosana, Sousa, Pedro Miguel, and Sampaio, André. Enterprise Transformation: The Serasa Experian Case. In Practice-Driven Research on Enterprise Transformation. Springer Berlin Heidelberg, 2011.

(61)

(62)

47

8. Appendix

A. List of Open Standards

Open Standard

Classification

SQL Obligatory

PNG Recommended

SVG Obligatory

XML Obligatory

XSLT 2.0 Obligatory

XSD Obligatory

XSL 1.1 Obligatory

XMPP Obligatory

UTF-8 Obligatory

ODF 1.1 Obligatory

PDF 1.7 Obligatory

XML 1.0 Obligatory

HTML 4.01 Obligatory

ATOM 1.0 Obligatory

CalDav Obligatory

CSS 2.1 Obligatory

HTTP/1.1 Obligatory

HTTPS Obligatory

Javascript 1.5 Recommended

WCAG 2.0 Obligatory /Recommended

WebDAV Recommended

RTSP Obligatory

IMAP 4 Obligatory

MIME Obligatory

POP3 Obligatory

POP3S Recommended

IMAPS Recommended

SMTP Obligatory

SMTPS Recommended

WCS Obligatory

WFS Obligatory

WMS Obligatory

WPS Obligatory

IPv6 Recommended

TLS 1.0 Obligatory

BPMN 2.0 Recommended

LDAP Obligatory

SAML 2.0 Obligatory

SOAP 1.1 Obligatory

WS-Addressing 1.0 Obligatory

WS-RM 1.1 Recommended

WS-Security 1.2 Recommended

(63)

(64)

(65)

Dashboard for evaluating the architecture of ICT projects in the Portuguese Public Administration