xand.es2021-12-15T21:13:54+00:00https://xand.esAlex VavilinSpring Security configuration for REST applications (I)2021-11-18T00:00:00+00:00https://xand.es/2021/11/18/configuring-spring-security<h3>Summary</h3>
<p>The main purpose of this implementation is to secure an application using Spring Boot. We assume that
the application uses <a href="https://jwt.io/">JWT</a> for the purposes of user's authentication. Please,
keep in mind that this is a very basic authentication/authorization mechanism designed for demonstration
purposes only. In production, you should use more sofisticated mechanisms, like persistent repositories, etc.</p>
<p>Basically, for each request there will be an instance of <code>RequestContext</code> class accessible by
<code>RequestContextHolder</code> and associated to a thread processing the request. This object will contain all the
information about the user and the JWT generated. Since it does not keep any dependency with Spring
Security framework you may access this information at any point inside your code.</p>
<p>When our application receives a request, <code>JWTTokenFilter</code> comes into play. This class extends Spring Security's
<code>OncePerRequestFilter</code>, so it filters all the incoming request. The main role of this filter is to verify
JWT provided by the user, e.g. valid signature, not expired, etc. Also, it keeps track of revoked tokens
(after user logout) using <code>JWTRepository</code>.</p>
<p><code>AccessDeniedHandlerJWT</code> is a handler which processes the 403 HTTP Status code returning an instance of
<code>Error403</code> class when the user performing the request lacks privileges for accesing resource. Basically
this handler comes into play when user does not pass <code>@PreAuthorize</code> constraint. </p>
<p><code>AuthenticationEntryPointJWT</code> is a class which implements <code>AuthenticationEntryPoint</code> interface from
Spring Security framework. The logic contained in this handler is fired when user performs a bad login, e.g.
invalid username/password pair and there is a need to return 401 HTTP Status code. An object of class
<code>Error401</code> will contain the information returned to user.</p>
<p>Finally a controller <code>AuthApiImpl</code> will perform a user login checking username/password pair. This controller
is also in charge of user's logout process update revoked tokens database.</p>
<p>On the following diagram you may see an overview of components involved in the implementation:</p>
<div class="nofloat noborder">
<img src="/public/img/2021-11-18-configuring-spring-security/security-config.png">
</div>
<h3>AppRunner</h3>
<p>This is the main class for application startup. The only special thing about this class are the annotations.
Especially note the contents of <code>@ComponentScan</code> annotation since this are the packages where our
Spring Boot application are going to find component definitions.</p>
<pre><code class="java">
package com.xand.authorizer;
import org.openapitools.jackson.nullable.JsonNullableModule;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.ExitCodeGenerator;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
@SpringBootApplication
@ComponentScan(basePackages = {
"com.xand.authorizer.oas.api",
"com.xand.authorizer.config",
"com.xand.authorizer.advice",
"com.xand.authorizer.filter"
})
public class AppRunner implements CommandLineRunner {
@Override
public void run(String... arg0) throws Exception {
if (arg0.length > 0 && arg0[0].equals("exitcode")) {
throw new ExitException();
}
}
public static void main(String[] args) throws Exception {
new SpringApplication(AppRunner.class).run(args);
}
static class ExitException extends RuntimeException implements ExitCodeGenerator {
private static final long serialVersionUID = 1L;
@Override
public int getExitCode() {
return 10;
}
}
@Bean
public WebMvcConfigurer webConfigurer() {
return new WebMvcConfigurer() {
};
}
@Bean
public com.fasterxml.jackson.databind.Module jsonNullableModule() {
return new JsonNullableModule();
}
}
</code></pre>
<h3>SecurityConfig</h3>
<p>Once <code>AppRunner</code> is defined we need to perform Spring Security configuration. <code>SecurityConfig</code> is the class
responsible for the whole configuration of our security framework.</p>
<p>Note the following:</p>
<ul>
<li><p><code>@EnableGlobalMethodSecurty</code> annotation with <code>prePostEnabled=true</code>. This allows us to use
<code>@PreAuthorize</code> annotations on controller's methods for fine-grained access based on roles or any
valid SpEL expression.</p></li>
<li><p>The class extends <code>WebSecurityConfigurerAdapter</code>.</p></li>
<li><p>We allow unrestricted access for anything under <code>/api/v1/public</code> (GET, POST) path and we require
user to be authenticated for anything else. Also we allow all Swagger stuff to be accesible without
authentication, e.g. <code>/swagger-ui/**</code>, <code>/swagger-resources/**</code>, <code>/api-docs</code>.</p></li>
</ul>
<p>Alongside the global security configuration this class instantiates the following components:</p>
<ul>
<li><code>UserDetailsService</code>. This is the repository for user's information. For development purposes we will use
a simple <code>InMemoryUserDetailsManager</code>. In order to use it you should define in your <code>application-dev.properties</code> something like:</li>
</ul>
<pre><code class="java">
# ------------------------------------------------------
# Security configuration (dev)
# This configuration only applies in development
# ------------------------------------------------------
app.security.user-details.users=admin;user;nobody
app.security.user-details.passwords=admin;user;nobody
app.security.user-details.roles=ADMIN;USER;NOBODY
</code></pre>
<p>Obviously you will need to use a real repository in production.</p>
<ul>
<li><p><code>PasswordEncoder</code>. BCrypt password encoder.</p></li>
<li><p><code>JWTTokenRepository</code>. An object of this class keeps track of all issued/revoked JWT. This class is a
<code>Runnable</code> and it executes in a separate <code>Thread</code> in order to perform cleanup of expired tokens.</p></li>
</ul>
<pre><code class="java">
package com.xand.authorizer.config;
import com.xand.authorizer.advice.AuthenticationEntryPointJWT;
import com.xand.authorizer.filter.JWTTokenFilter;
import com.xand.authorizer.util.Constants;
import com.xand.authorizer.util.JWTRepository;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.env.Environment;
import org.springframework.http.HttpMethod;
import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
import org.springframework.security.config.annotation.method.configuration.EnableGlobalMethodSecurity;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
import org.springframework.security.config.http.SessionCreationPolicy;
import org.springframework.security.core.userdetails.User;
import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.security.core.userdetails.UserDetailsService;
import org.springframework.security.crypto.bcrypt.BCryptPasswordEncoder;
import org.springframework.security.crypto.password.PasswordEncoder;
import org.springframework.security.provisioning.InMemoryUserDetailsManager;
import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;
import org.springframework.security.web.authentication.www.BasicAuthenticationFilter;
import org.springframework.web.cors.CorsConfiguration;
import org.springframework.web.cors.UrlBasedCorsConfigurationSource;
import org.springframework.web.filter.CorsFilter;
import java.util.Arrays;
@Configuration
@EnableGlobalMethodSecurity(prePostEnabled = true)
public class SecurityConfig extends WebSecurityConfigurerAdapter {
private Logger log = LoggerFactory.getLogger(SecurityConfig.class);
@Value("${app.security.user-details.users}")
private String devUsersProperties;
@Value("${app.security.user-details.passwords}")
private String devPasswordsProperties;
@Value("${app.security.user-details.roles}")
private String devRolesProperties;
private final JWTTokenFilter jwtTokenFilter;
private final AuthenticationEntryPointJWT authenticationEntrypointJWT;
private final Environment environment;
public SecurityConfig(@Autowired final JWTTokenFilter jwtTokenFilter,
@Autowired final AuthenticationEntryPointJWT authenticationEntrypointJWT,
@Autowired final Environment environment) {
this.jwtTokenFilter = jwtTokenFilter;
this.authenticationEntrypointJWT = authenticationEntrypointJWT;
this.environment = environment;
}
@Override
protected void configure(HttpSecurity http) throws Exception {
// Enable CORS and disable CSRF
http = http.cors().and().csrf().disable();
http = http.httpBasic().disable().formLogin().disable();
// set 401/403 handlers
http = http
.exceptionHandling().authenticationEntryPoint(this.authenticationEntrypointJWT)
.and();
// Set session management to stateless
http = http
.sessionManagement()
.sessionCreationPolicy(SessionCreationPolicy.STATELESS)
.and();
// Set permissions on endpoints
http.authorizeRequests()
// Our public endpoints
.antMatchers(HttpMethod.GET,
"/swagger-ui/**",
"/swagger-resources/**",
"/api-docs", "/api/v1/public/**" ).permitAll()
.antMatchers(HttpMethod.POST,
"/api/v1/public/**", "/api/v1/auth" ).permitAll()
// Our private endpoints
.anyRequest().authenticated();
// Add JWT token filter
http.addFilterBefore(
jwtTokenFilter,
UsernamePasswordAuthenticationFilter.class
);
}
@Override
protected void configure(AuthenticationManagerBuilder auth) throws Exception {
auth.userDetailsService(this.userDetailsService());
}
@Bean
@Override
public UserDetailsService userDetailsService() {
UserDetailsService output = null;
if(this.isDevEnvironment()) {
InMemoryUserDetailsManager userDetailsManager = new InMemoryUserDetailsManager();
// build development UserDetailsService
String[] users = StringUtils.split(this.devUsersProperties, ";");
String[] passwords = StringUtils.split(this.devPasswordsProperties, ";");
String[] roles = StringUtils.split(this.devRolesProperties, ";");
for(int i = 0; i < users.length; i++) {
String user = users[i];
String password = passwords[i];
String role = roles[i];
UserDetails userDetails = User.withUsername(user)
.passwordEncoder(passwordEncoder()::encode)
.password(password)
.roles(role)
.build();
userDetailsManager.createUser(userDetails);
}
output = userDetailsManager;
} else {
// build a real UserDetailsService
}
return output;
}
@Bean
public PasswordEncoder passwordEncoder() {
return new BCryptPasswordEncoder();
}
@Bean
public JWTRepository jwtRevokeRepository() {
JWTRepository output = new JWTRepository();
return output;
}
private boolean isDevEnvironment() {
String[] activeProfiles = this.environment.getActiveProfiles();
return Arrays.asList(activeProfiles).contains(Constants.ENVIRONMENT_DEV);
}
}
</code></pre>
<h3>AccessDeniedHandlerJWT</h3>
<p>This component handles 403 (Forbidden) HTTP status. When Spring Security detects that user tries to access
a resource which is not allowed to, for example via <code>@PreAuthorize</code> annotation it will throw
<code>AccessDeniedException</code>. Since this class is annotated with <code>@ControllerAdvice</code> it will intercept the exception
thanks to <code>@ExceptionHandler</code> annotation and send the appropriate response.</p>
<p>Also, when you want to deny access to some resource inside your code you may throw <code>AccessDeniedException</code>
in order to prevent the access and automatically return 403 HTTP status code.</p>
<pre><code class="java">
package com.xand.authorizer.advice;
import com.xand.authorizer.exception.AppAccessDeniedException;
import com.xand.authorizer.oas.model.Error403;
import com.xand.authorizer.util.RequestContext;
import com.xand.authorizer.util.RequestContextHolder;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.security.access.AccessDeniedException;
import org.springframework.web.bind.annotation.ControllerAdvice;
import org.springframework.web.bind.annotation.ExceptionHandler;
import javax.servlet.http.HttpServletRequest;
import java.text.SimpleDateFormat;
import java.util.Date;
/**
* This class treats 403 error. This means that we know who the user is, but the access is
* explicitly forbidden
*/
@ControllerAdvice
public class AccessDeniedHandlerJWT {
private static final SimpleDateFormat TSTAMP_FORMAT = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
@ExceptionHandler(AccessDeniedException.class)
public ResponseEntity<Error403> handle(AccessDeniedException ex, HttpServletRequest request) {
RequestContext requestContext = RequestContextHolder.get();
String user = requestContext.getJwt().getSubject();
Error403 err = this.buildError403(request, user, ex.getMessage());
ResponseEntity<Error403> output = new ResponseEntity<>(err, HttpStatus.FORBIDDEN);
return output;
}
@ExceptionHandler(AppAccessDeniedException.class)
public ResponseEntity<Error403> handle(AppAccessDeniedException ex, HttpServletRequest request) {
Error403 err = this.buildError403(request, null, ex.getMessage());
ResponseEntity<Error403> output = new ResponseEntity<>(err, HttpStatus.FORBIDDEN);
return output;
}
private Error403 buildError403(HttpServletRequest request, String user, String exceptionMessage) {
String queryString = request.getQueryString();
if(queryString == null) {
queryString = "";
}
String method = request.getMethod();
String requestUri = request.getRequestURI();
Error403 err = new Error403();
err.setTimestamp(TSTAMP_FORMAT.format(new Date()));
if(user != null) {
err.setError("Access is denied for user " + user);
} else {
err.setError(exceptionMessage);
}
err.setQueryString(queryString);
err.setMethod(method);
err.setRequestUri(requestUri);
return err;
}
}
</code></pre>
<h3>AuthenticationEntryPointJWT</h3>
<p>This class acts as an interceptor for failed login attempts. Basically when no JWT token is provided in the
<code>Authorization</code> header and, as a consequence, Spring Security is unable to build <code>SecurityContextHandler</code>
it will return 401 HTTP status. After that, this class will intercept the request in order to build
appropriate response.</p>
<pre><code class="java">
package com.xand.authorizer.advice;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.xand.authorizer.oas.model.Error401;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
import org.springframework.security.core.AuthenticationException;
import org.springframework.security.web.AuthenticationEntryPoint;
import org.springframework.stereotype.Component;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.Date;
/**
* This class treats 401 error when no token is provided.
*/
@Component
public class AuthenticationEntryPointJWT implements AuthenticationEntryPoint {
private static final SimpleDateFormat TSTAMP_FORMAT = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
@Autowired
private ObjectMapper objectMapper;
@Override
public void commence(HttpServletRequest request,
HttpServletResponse response,
AuthenticationException ex) throws IOException {
String queryString = request.getQueryString();
if(queryString == null) {
queryString = "";
}
String method = request.getMethod();
String requestUri = request.getRequestURI();
Error401 err = new Error401();
err.setTimestamp(TSTAMP_FORMAT.format(new Date()));
err.setError("You need to perform a full authentication before accesing this resource");
err.setQueryString(queryString);
err.setMethod(method);
err.setRequestUri(requestUri);
response.setContentType(MediaType.APPLICATION_JSON_VALUE);
response.setStatus(HttpServletResponse.SC_UNAUTHORIZED);
this.objectMapper.writeValue(response.getOutputStream(), err);
}
}
</code></pre>
<h3>JWTTokenFilter</h3>
<p>This class is a filter so it extends <code>OncePerRequestFilter</code>. This filter is responsible for building
<code>SecurityContextHolder</code>. It intercepts all HTTP requests, looks for JWT within the <code>Authorization</code> header,
parses it and performs the checks, e.g. signature, expiration, etc.</p>
<p>In case no token is provided inside the <code>Authorization</code> header or the token provided is invalid (expired,
not valid signature, etc) the flow continues as normal until it hits Spring Security internals which makes
the decision to allow the request (endpoints avaialble publicly) or to return 403 HTTP Status.</p>
<pre><code class="java">
package com.xand.authorizer.filter;
import com.xand.authorizer.util.JWT;
import com.xand.authorizer.util.JWTRepository;
import com.xand.authorizer.util.RequestContext;
import com.xand.authorizer.util.RequestContextHolder;
import io.jsonwebtoken.*;
import io.jsonwebtoken.security.Keys;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpHeaders;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.security.core.userdetails.UserDetailsService;
import org.springframework.security.web.authentication.WebAuthenticationDetailsSource;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import javax.crypto.SecretKey;
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.time.OffsetDateTime;
import java.time.ZoneOffset;
import java.util.Date;
import java.util.List;
@Component
public class JWTTokenFilter extends OncePerRequestFilter {
@Value("${app.security.jws.secretKey}")
private String jwsSecret;
@Autowired
private UserDetailsService userDetailsService;
@Autowired
private JWTRepository jwtRepository;
@Override
protected void doFilterInternal(HttpServletRequest request,
HttpServletResponse response,
FilterChain filterChain)
throws ServletException, IOException {
// Get authorization header and validate
final String header = request.getHeader(HttpHeaders.AUTHORIZATION);
if (StringUtils.isEmpty(header) || !header.startsWith("Bearer ")) {
filterChain.doFilter(request, response);
return;
}
// Get jwt token and validate
String token = header.substring("Bearer ".length());
token = token.trim();
SecretKey secretKey = Keys.hmacShaKeyFor(this.jwsSecret.getBytes());
String username;
String tokenId;
OffsetDateTime expiration;
OffsetDateTime issuedAt;
String issuer;
try {
Jws<Claims> claimsJws = Jwts.parserBuilder().setSigningKey(secretKey).build()
.parseClaimsJws(token);
// at this point we also check that the token is not expired
username = claimsJws.getBody().getSubject();
tokenId = claimsJws.getBody().getId();
issuer = claimsJws.getBody().getIssuer();
Date exp = claimsJws.getBody().getExpiration();
expiration = exp.toInstant().atOffset(ZoneOffset.UTC);
Date dIssuedAt = claimsJws.getBody().getIssuedAt();
issuedAt = dIssuedAt.toInstant().atOffset(ZoneOffset.UTC);
} catch(JwtException e) {
filterChain.doFilter(request, response);
return;
}
// Check that token is not revoked
boolean revoked = this.jwtRepository.isTokenRevoked(tokenId);
if(revoked) {
filterChain.doFilter(request, response);
return;
}
// Get user identity and set it on the spring security context
UserDetails userDetails = this.userDetailsService.loadUserByUsername(username);
UsernamePasswordAuthenticationToken
authentication = new UsernamePasswordAuthenticationToken(
userDetails, null,
userDetails == null ?
List.of() : userDetails.getAuthorities()
);
authentication.setDetails(
new WebAuthenticationDetailsSource().buildDetails(request)
);
SecurityContextHolder.getContext().setAuthentication(authentication);
// create JWT object
JWT jwt = new JWT();
jwt.setSubject(username);
jwt.setExpiration(expiration);
jwt.setId(tokenId);
jwt.setRawToken(token);
jwt.setIssuer(issuer);
jwt.setIssuedAt(issuedAt);
String remoteIp = request.getRemoteAddr();
String userAgent = request.getHeader("User-Agent");
// create and populate RequestContext object
RequestContext requestContext = new RequestContext();
requestContext.setJwt(jwt);
requestContext.setRemoteIp(remoteIp);
requestContext.setRemoteUserAgent(userAgent);
userDetails.getAuthorities().forEach(it -> {
requestContext.getAuthorities().add(it.getAuthority());
});
RequestContextHolder.set(requestContext);
filterChain.doFilter(request, response);
}
}
</code></pre>
<h3>AuthApiImpl</h3>
<p>This class is responsible for user authentication and logout. </p>
<p>When authentication is performed this class simply verifies username/password and generates JWT. If
username/password pair is invalid and <code>UnathorizedException</code> is thrown.</p>
<p>On the other side, when logout is performed the class simply informs <code>JWTRepository</code> that the JWT is revoked.</p>
<pre><code class="java">
package com.xand.authorizer.oas.api;
import com.xand.authorizer.exception.UnathorizedException;
import com.xand.authorizer.oas.model.AuthUserRequest;
import com.xand.authorizer.oas.model.AuthUserResponse;
import com.xand.authorizer.oas.model.Error401;
import com.xand.authorizer.oas.model.LogoutResponse;
import com.xand.authorizer.util.*;
import io.jsonwebtoken.Jwts;
import io.jsonwebtoken.security.Keys;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.security.core.userdetails.UserDetailsService;
import org.springframework.security.crypto.password.PasswordEncoder;
import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestController;
import javax.annotation.PostConstruct;
import javax.crypto.SecretKey;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.validation.Valid;
import java.text.SimpleDateFormat;
import java.time.OffsetDateTime;
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.util.*;
@RestController
public class AuthApiImpl implements AuthApi {
private static final SimpleDateFormat TSTAMP_FORMAT = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
private Logger log = LoggerFactory.getLogger(AuthApiImpl.class);
private PasswordEncoder passwordEncoder;
private UserDetailsService userDetailsService;
private JWTRepository jwtRepository;
@Value("${app.security.jws.secretKey}")
private String jwsSecret;
@Value("${app.security.jws.issuer}")
private String jwsIssuer;
@Value("${app.security.jws.validSeconds}")
private String sJwsValidSeconds;
private int jwsValidSeconds;
public AuthApiImpl(UserDetailsService userDetailsService,
PasswordEncoder passwordEncoder,
JWTRepository jwtRepository) {
this.userDetailsService = userDetailsService;
this.passwordEncoder = passwordEncoder;
this.jwtRepository = jwtRepository;
}
@PostConstruct
public void init() {
this.jwsValidSeconds = Integer.parseInt(sJwsValidSeconds);
}
@Override
public ResponseEntity<AuthUserResponse> authUser(@Valid AuthUserRequest authUserRequest) {
RequestContext requestContext = RequestContextHolder.get();
AuthUserResponse res;
String jws;
if(requestContext == null) {
// normal user login
String username = authUserRequest.getUsername().trim();
String password = authUserRequest.getPassword().trim();
UserDetails userDetails;
try {
userDetails = this.userDetailsService.loadUserByUsername(username);
} catch(Throwable t) {
// exit immediately
throw new UnathorizedException(username, Constants.MSG_LOGIN_FAILED, false);
}
String encodedPassword = userDetails.getPassword();
boolean passwordMatch = false;
try {
passwordMatch = this.passwordEncoder.matches(password, encodedPassword);
} catch(Throwable t) {
log.error("An error occured", t);
}
if(!passwordMatch) {
// exit immediately
// additionally you shoud report this attempt to the audit,
// since it may be a brute force attack the user provided
throw new UnathorizedException(username, Constants.MSG_LOGIN_FAILED, true);
}
List<String> sAuthorities = new ArrayList<>();
Collection<? extends GrantedAuthority> authorities = userDetails.getAuthorities();
for(GrantedAuthority ga : authorities) {
String authority = ga.getAuthority();
sAuthorities.add(authority);
}
// TODO: Do something with variables below
boolean accountNonExpired = userDetails.isAccountNonExpired();
boolean accountNonLocked = userDetails.isAccountNonLocked();
boolean credentialsNonExpired = userDetails.isCredentialsNonExpired();
boolean enabled = userDetails.isEnabled();
OffsetDateTime issuedAt = OffsetDateTime.now(ZoneOffset.UTC);
OffsetDateTime expiration = issuedAt.plusSeconds(this.jwsValidSeconds);
SecretKey secretKey = Keys.hmacShaKeyFor(this.jwsSecret.getBytes());
String tokenId = UUID.randomUUID().toString();
String rawToken = Jwts.builder()
.setIssuer(this.jwsIssuer)
.setSubject(username)
.setId(tokenId)
.claim("authorities", sAuthorities)
.setIssuedAt(Date.from(issuedAt.toInstant()))
.setExpiration(Date.from(expiration.toInstant()))
.signWith(secretKey)
.compact();
jws = "Bearer " + rawToken;
res = new AuthUserResponse();
res.setAccessToken(jws);
res.setIssuer(this.jwsIssuer);
res.setSubject(username);
res.setAuthorities(sAuthorities);
res.setIssuedAt(issuedAt);
res.setExpiration(expiration);
// save generated token into repository
JWT jwt = new JWT();
jwt.setId(tokenId);
jwt.setExpiration(expiration);
jwt.setSubject(username);
jwt.setRawToken(rawToken);
this.jwtRepository.put(jwt);
} else {
// the user is already authenticated
// return the original token
jws = "Bearer " + requestContext.getJwt().getRawToken();
List<String> authorities = new ArrayList<>();
requestContext.getAuthorities().forEach(it -> {
authorities.add(it);
});
res = new AuthUserResponse();
res.setAccessToken(jws);
res.setIssuer(requestContext.getJwt().getIssuer());
res.setSubject(requestContext.getJwt().getSubject());
res.setAuthorities(authorities);
res.setIssuedAt(requestContext.getJwt().getIssuedAt());
res.setExpiration(requestContext.getJwt().getExpiration());
}
return ResponseEntity.ok().header(HttpHeaders.AUTHORIZATION, jws).body(res);
}
@Override
public ResponseEntity<LogoutResponse> revokeJwt() {
RequestContext requestContext = RequestContextHolder.get();
JWT jwt = requestContext.getJwt();
this.jwtRepository.revoke(jwt);
LogoutResponse res = new LogoutResponse();
res.setUsername(jwt.getSubject());
return new ResponseEntity<>(res, HttpStatus.OK);
}
@ExceptionHandler(value = UnathorizedException.class)
public ResponseEntity<Error401> handle401(UnathorizedException ex,
HttpServletRequest request) {
String queryString = request.getQueryString();
if(queryString == null) {
queryString = "";
}
String method = request.getMethod();
String requestUri = request.getRequestURI();
Error401 err = new Error401();
err.setTimestamp(TSTAMP_FORMAT.format(new Date()));
err.setError(ex.getMessage());
err.setQueryString(queryString);
err.setMethod(method);
err.setRequestUri(requestUri);
ResponseEntity<Error401> output = new ResponseEntity<>(err, HttpStatus.UNAUTHORIZED);
return output;
}
}
</code></pre>
<h3>JWTRepository</h3>
<p>An object of this class is used to keep track of all issued JWT. Also, when user performs a logout we
revoke the token used by the user and we keep it inside a collection of revoked tokens.</p>
<p>This class is a <code>Runnable</code>. This way we may perform a periodic cleanup of expired tokens.</p>
<pre><code class="java">
package com.xand.authorizer.util;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.time.Instant;
import java.time.OffsetDateTime;
import java.util.*;
/**
* This class keeps track of issued and revoked token. It is a Runnable, so it executes in a separate thread.
* This way we may perioridically check and remove expired tokens.
*/
public class JWTRepository implements Runnable {
private static final Logger log = LoggerFactory.getLogger(JWTRepository.class);
private final Map<String, JWT> issued = new HashMap<>();
private final Map<String, JWT> revoked = new HashMap<>();
private Thread thread;
private boolean running = true;
public JWTRepository() {
this.thread = new Thread(this);
this.thread.start();
}
public void put(JWT jwt) {
synchronized (this.issued) {
this.issued.put(jwt.getId(), jwt);
}
}
public JWT get(String id) {
return this.issued.get(id);
}
public boolean isTokenRevoked(String id) {
return this.revoked.containsKey(id);
}
public void revoke(JWT jwt) {
synchronized (this.revoked) {
this.revoked.put(jwt.getId(), jwt);
}
}
public void run() {
while(this.running) {
List<String> issuedToRemove = new ArrayList<>();
List<String> revokedToRemove = new ArrayList<>();
synchronized (this.issued) {
Collection<JWT> issuedCol = this.issued.values();
for(JWT jwt : issuedCol) {
OffsetDateTime expiration = jwt.getExpiration();
if(expiration.toInstant().isAfter(Instant.now())) {
issuedToRemove.add(jwt.getId());
}
}
for(String id : issuedToRemove) {
this.issued.remove(id);
}
}
synchronized (this.revoked) {
Collection<JWT> revokedCol = this.revoked.values();
for(JWT jwt : revokedCol) {
OffsetDateTime expiration = jwt.getExpiration();
if(expiration.toInstant().isAfter(Instant.now())) {
revokedToRemove.add(jwt.getId());
}
}
for(String id : revokedToRemove) {
this.revoked.remove(id);
}
}
try {
Thread.sleep(5000);
} catch(InterruptedException e){
log.error("An error occured", e);
}
}
}
}
</code></pre>
<h3>KeyApiImpl</h3>
<p>This is just an example class which demonstrates the use of <code>@PreAuthorize</code> annotation. In this case, only
users with <code>ROLE_ADMIN</code> are allowed to access the resource, other roles will get 403 Forbidden HTTP status
code.</p>
<p>This class also demonstrates the use of <code>RequestContext</code> which is used to get information about current
user, such as authorities and username (subject in terms of JWT).</p>
<pre><code class="java">
package com.xand.authorizer.oas.api;
import com.xand.authorizer.oas.model.KeySlotRead;
import com.xand.authorizer.oas.model.KeySlotWrite;
import com.xand.authorizer.util.RequestContext;
import com.xand.authorizer.util.RequestContextHolder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.web.bind.annotation.RestController;
import javax.validation.Valid;
@RestController
public class KeyApiImpl implements KeyApi {
private static final Logger log = LoggerFactory.getLogger(KeyApiImpl.class);
@Override
@PreAuthorize("hasAuthority('ROLE_ADMIN')")
public ResponseEntity<KeySlotRead> postKey(@Valid KeySlotWrite keySlotWrite) {
RequestContext requestContext = RequestContextHolder.get();
log.debug("USER => {}", requestContext.getJwt().getSubject());
KeySlotRead res = new KeySlotRead();
ResponseEntity<KeySlotRead> output = new ResponseEntity<>(res, HttpStatus.CREATED);
return output;
}
}
</code></pre>
<h3>UnathorizedException</h3>
<p>This exception is thrown when a failed attempt occurs.</p>
<p>This object holds the following information:</p>
<ul>
<li><p><code>username</code>. Username provided during authentication process.</p></li>
<li><p><code>message</code>. Message shown to the user performing the request.</p></li>
<li><p><code>userExists</code>. This boolean attribute indicates that the user actually exists in the user repository. This
attribute may be used in order to detect that user's account is under brute force attack. Additionally you
should be sure to not provide this attribute to the outside, since it may leave to a security breach.
Basically, if you provide this attribute to an attacker you are disclosing that the account actually exists
in our system.</p></li>
</ul>
<pre><code class="java">
package com.xand.authorizer.exception;
/**
* Exception holding information about failed login.
*
* IMPORTANT: You MUST never show the value of userExists property to the client since it may
* leave to a security breach.
*/
public class UnathorizedException extends RuntimeException {
private final String username;
private final boolean userExists;
public UnathorizedException(final String username,
final String message,
final boolean userExists) {
super(message);
this.username = username;
this.userExists = userExists;
}
public String getUsername() {
return this.username;
}
public boolean isUserExists() {
return this.userExists;
}
}
</code></pre>
<h3>AppAccessDeniedException</h3>
<p>This is a <code>RuntimeException</code> which is thrown by the business layer in order to forbid the access to a
resource. We use a separate exception for this task to not keep dependecy with Spring Security framework.</p>
<pre><code class="java">
package com.xand.authorizer.exception;
public class AppAccessDeniedException extends RuntimeException {
public AppAccessDeniedException(final String message) {
super(message);
}
}
</code></pre>
<h3>RequestContext and RequestContextHolder</h3>
<p><code>RequestContextHolder</code> is just a class responsible for bounding <code>RequestContext</code> to a current <code>Thread</code> using
<code>ThreadLocal</code>. In order to get the <code>RequestContext</code> object we may issue a call to <code>RequestContextHolder.get()</code>
at any point in our code.</p>
<p><code>RequestContext</code> object holds all the security related information associated to a request. We use a custom
class in order to avoid the dependency with Spring framework, so we can use this class anywhere inside
the code.</p>
<pre><code class="java">
package com.xand.authorizer.util;
public class RequestContextHolder {
private static final ThreadLocal<RequestContext> context = new ThreadLocal<>();
public static void set(RequestContext requestContext) {
context.set(requestContext);
}
public static RequestContext get() {
return context.get();
}
public static void clean() {
context.remove();
}
}
</code></pre>
<pre><code class="java">
package com.xand.authorizer.util;
import lombok.Getter;
import lombok.Setter;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;
@Getter
@Setter
public class RequestContext implements Serializable {
private JWT jwt;
private String remoteIp;
private String remoteUserAgent;
private List<String> authorities = new ArrayList<>();
}
</code></pre>
<h3>Error401 and Error403</h3>
<p>These classes are simple models which contain information about the error. They are generated using Swagger
and they are part of the API Rest specification.</p>
<h3>pom.xml</h3>
<p>This pom.xml lists all necessary dependencies.</p>
<pre><code class="xml">
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.xand.authorizer</groupId>
<artifactId>ov-authorizer-backend</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.5.6</version>
<relativePath/>
</parent>
<properties>
<java.version>11</java.version>
<springboot-version>2.5.6</springboot-version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>${springboot-version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
<version>${springboot-version}</version>
</dependency>
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt-api</artifactId>
<version>0.11.2</version>
</dependency>
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt-impl</artifactId>
<version>0.11.2</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>io.jsonwebtoken</groupId>
<artifactId>jjwt-jackson</artifactId> <!-- or jjwt-gson if Gson is preferred -->
<version>0.11.2</version>
<scope>runtime</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.0</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<version>2.2.1.RELEASE</version>
</plugin>
</plugins>
</build>
</project>
</code></pre>
Git: haciendo releases2016-12-13T00:00:00+00:00https://xand.es/2016/12/13/releasing-with-git<h3>Introducción</h3>
<p>Esto es una forma de preparar las releases de software cuando el equipo de desarrollo sigue la metodología Gitflow (o alguna de sus variantes).
Llevo utilizando esta forma de preparar las releases desde hace algunos años y los resultados son bastante positivos.</p>
<p>El método cuenta con 6 pasos. Para el propósito de este manual vamos a suponer que la versión que se prentende generar es la 1.7.
Al mismo tiempo se entiende que ya se tiene la versión 1.6 generada anteriormente.</p>
<h3>Rama: integración</h3>
<p>Nuestro punto de partida para la generación de la release tiene que ser justo la release anterior, es decir, el commit marcado con el tag v.1.6.
Para saber cuál es el commit utilizaremos el siguiente comando:</p>
<pre><code class="console">
$ git log v.1.6
</code></pre>
<p>En la salida de dicho comando vamos a ver el historial de los commits, el que nos interesa estará como el primero, en este caso:</p>
<pre><code class="console">
commit <strong>01758f6172503abaf8abd91838a849b8080604e5</strong> (tag: v.1.6)
Merge: 9d47987f d0189b7b
Author: Alex Vavilin <xand@xand.es>
Date: Wed Dec 11 13:57:50 2016 +0000
Merge branch 'integration/v.1.6' into 'master'
Integration/v.1.6
commit d0189b7b29a0c836ee53193f5b15d6c2cff4fc1c
Author: Alex Vavilin <xand@xand.es>
Date: Wed Dec 11 13:57:49 2016 +0000
[FEAT#790] Descripción de la feature
</code></pre>
<p>En este caso el commit que nos interesa es el <strong>01758f61</strong>, es decir, el primero que aparece. Para crear la rama para la integración se tiene que crear una rama con el nombre <strong>integration/v.1.7</strong> a partir de este commit. Se hace con el siguiente comando:</p>
<pre><code class="console">
$ git checkout -b integration/v.1.7 01758f61
</code></pre>
<p>Con esto ya tenemos la rama donde prepararemos la release.</p>
<h3>Elección de los commits y su mergeo</h3>
<p>El siguiente paso consiste en determinar cuáles son los commits que se quieren incluir en la release. </p>
<pre><code class="console">
$ git log integration/v.1.7..develop --oneline --merges --first-parent
</code></pre>
<p>Con este comando vamos a obtener una lista de commits que han sido mergeados en la rama <strong>develop</strong> y no han sido llevados a la release anterior, esto es, no están presentes en el tag <strong>v.1.6</strong>. Podemos exportar la lista y escoger únicamente aquellos commits que nos interesan para la release.</p>
<p>Es importante tener en cuenta que cada commit posterior lleva integrado también el anterior, con lo cual, escoger el primer commit de la lista (viendo desde arriba) equivale a escoger todos.</p>
<h3>Despliegue en el entorno de pruebas</h3>
<h3>Fixes para la corrección de fallos</h3>
<h3>Merge a master</h3>
<h3>Merge a develop</h3>
<ol>
<li>Se crea la rama integration/v.X.Y a partir del master o bien a partir
de un commit (el último).</li>
<li>Se cogen todos los commits del develop a la rama en cuestión.</li>
<li>Se realiza el despliegue de la rama en entorno staging.</li>
<li>Se realizan los FIX# para la corrección de fallos.</li>
<li>Se hace merge integration/v.X.Y -> master y se genera el tag v.X.Y.
(A partir de aquí se puede generar una nueva versión).</li>
<li>Se hace merge develop -> integration/v.X.Y</li>
<li>Se hace merge integration/v.X.Y -> develop.</li>
<li>Se elimina la rama integration/v.X.Y.</li>
</ol>
Jenkins: sending email from a file2016-11-05T00:00:00+00:00https://xand.es/2016/11/05/jenkins-email-from-file<p>Jenkins has a very cool plugin called <a href="https://wiki.jenkins.io/display/JENKINS/Email-ext+plugin">Editable Email Notifications</a>. However it is a bit complicated to send a good-looking complex email from the task configuration it offers.
Personally I find it easier to compose en email with an external script, write it to a file and then use the contents of this file as a body for email.</p>
<p>It seems pretty obvious, but the tricky part is actually reading the file into a variale for setting message's content. Fortunately the plugin offers the possiblity to execute Groovy script before sending the email (Pre-Send Script option) which may alter the content of the message body.</p>
<p>Here is the script:</p>
<pre><code class="groovy">
// RESULT\_EMAIL environment variable must contain file path where the body
// is stored
def emailContentFile = build.getEnvironment()['RESULT\_EMAIL'].trim()
def emailContent = ""
def f = new File(emailContentFile).eachLine {
line -> emailContent += "$line";
}
msg.setContent(emailContent,'text/html; charset=utf-8')
</code></pre>
<p>The only thing I need to mention is that this way the content of Default Content parameter is completely ignored.</p>
<p><strong>UPDATE:</strong> There is actually a better approach to this. You may simply provide somthing like this in as Default Content parameter:</p>
<pre><code class="console">
${FILE, path="result\_email.html"}
</code></pre>
<p>result_email.html should be relative to workspace.</p>
Liquibase context control2016-10-21T00:00:00+00:00https://xand.es/2016/10/21/liquibase-control-contexts<p>When using liquibase for database versioning it is very common to use the same approach to load test data.
For example, I use it to load data on development (developer's machines) and staging environment.
But, in any circumstances, the test data should not be loaded in any real environment (production).</p>
<p>This behavior may be achieved using <a href="https://www.liquibase.org/documentation/contexts.html">liquibase contexts</a>.</p>
<p>Imagine the following changesets, for example:</p>
<pre><code class="xml">
<!-- this changeset must only be executed in dev,staging environments -->
<changeSet id="1566987495224" author="developer@example.com" context="dev,staging">
<insert tableName="ov_emp_rel_employee_attribute">
<column name="employee_ref_id" value="9815160b-de4e-434b-9052-b54b357892db"/>
<column name="attribute_ref_id" value="f5f00fe2-281e-4d5b-b324-82604b5a5009"/>
<column name="value" value="pperez@example.com"/>
<column name="creation_date" valueComputed="CURRENT_TIMESTAMP"/>
<column name="updated_date" valueComputed="CURRENT_TIMESTAMP"/>
</insert>
</changeSet>
<!-- this changeset must always be executed -->
<changeSet id="1566987495225" author="developer@example.com">
<addColumn tableName="ov_dst_edition">
<column name="status" type="varchar(10)"/>
<column name="init_date" type="timestamp"/>
<column name="end_date" type="timestamp"/>
</addColumn>
</changeSet>
</code></pre>
<p>In this situation if the developer forgets to indicate the context (first change request in the example),
this changes will go into production environment. In order to avoid this situation I prepared a simple unit test which
perform the following checks:</p>
<ul>
<li>There are no duplicate ids in the liquibase xml files. This is not strictly necessary, but I prefer to have unique ids.</li>
<li>All liquibase statements capable of changing data must include the context information. See OPS_NOT_ALLOWED_WO_CONTEXT
array. The elements are following: insert, update, delete, sqlFile, sql, mergeColumns, loadUpdateData, loadData,
executeCommand, customChange.</li>
<li>Sometimes there are exceptions to the above rule (for example, a catalog table), in which case the change request must
be manually whitelisted (see ALLOWED_CHANGESETS array).</li>
</ul>
<p>The unit test is the following:</p>
<pre><code class="java">
package net.overlap.integration;
import org.apache.commons.io.FileUtils;
import org.dom4j.Document;
import org.dom4j.Element;
import org.dom4j.Node;
import org.dom4j.io.DOMReader;
import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import java.io.File;
import java.net.URL;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.List;
@RunWith(JUnit4.class)
public class LiquibaseContextTest {
private static final String[] EXCLUDED_FILES = {"changelog.xml"};
private static final String[] OPS_NOT_ALLOWED_WO_CONTEXT = {
"insert", "update", "delete", "sqlFile", "sql", "mergeColumns",
"loadUpdateData", "loadData", "executeCommand", "customChange"
};
private static final String[] ALLOWED_CHANGESETS = {
"1562773293100", "1562773293104"
};
@Test
public void testPolicyViolation() throws Exception {
List<ChangesetDTO> changesets = this.load();
boolean policyViolation = false;
for(ChangesetDTO c : changesets) {
boolean v = c.violatesPolicy(OPS_NOT_ALLOWED_WO_CONTEXT);
System.out.println(c.toString() + " -> "
+ String.valueOf(v));
if(v) {
if(Arrays.binarySearch(ALLOWED_CHANGESETS, c.id) < 0) {
policyViolation = true;
System.out.println(c.file + ":" + c.id + " -> "
+ "CONTEXT POLICY VIOLATION");
}
}
}
Assert.assertFalse(policyViolation);
}
@Test
public void testDuplicates() throws Exception {
List<ChangesetDTO> changesets = this.load();
boolean duplicates = hasDuplicateChangesets(changesets);
Assert.assertFalse(duplicates);
}
private List<ChangesetDTO> load() throws Exception {
// list all files in /database
URL url = Paths.get("target").toUri().toURL();
String sUrl = url.toString();
String sPath = sUrl.substring("file:".length());
sPath += "classes";
sPath += File.separator;
sPath += "database";
List<String> filesToCheck = new ArrayList<>();
Collection<File> files = FileUtils.listFiles(new File(sPath), new String[]{"xml"}, false);
for(File f : files) {
String filename = f.getName();
if(Arrays.binarySearch(EXCLUDED_FILES, filename) == -1) {
filesToCheck.add(filename);
}
}
String[] arrFilesToCheck = new String[filesToCheck.size()];
filesToCheck.toArray(arrFilesToCheck);
Arrays.sort(arrFilesToCheck);
List<ChangesetDTO> changesets = new ArrayList<>();
for(String f : arrFilesToCheck) {
String fullPath = sPath + File.separator + f;
System.out.println("Checking file: " + fullPath);
File xmlFile = new File(fullPath);
DocumentBuilderFactory dbf = DocumentBuilderFactory.newDefaultInstance();
DocumentBuilder dBuilder = dbf.newDocumentBuilder();
org.w3c.dom.Document doc = dBuilder.parse(xmlFile);
DOMReader domReader = new DOMReader();
Document document = domReader.read(doc);
this.processSingleFile(document, changesets, f);
}
return changesets;
}
private boolean hasDuplicateChangesets(List<ChangesetDTO> changesets) {
List<String> checkList = new ArrayList<>();
boolean output = false;
for(ChangesetDTO c : changesets) {
if(checkList.contains(c.id)) {
output = true;
System.out.println("Detected duplicate changeset: " + c.id);
} else {
checkList.add(c.id);
}
}
return output;
}
private void processSingleFile(Document document,
List<ChangesetDTO> changesets,
String file) {
List nodes = document.selectNodes("//databaseChangeLog/changeSet");
for(int i = 0; i < nodes.size(); i++) {
Node node = (Node)nodes.get(i);
Element elem = (Element)nodes.get(i);
String changeSetId = elem.attributeValue("id");
String context = elem.attributeValue("context");
ChangesetDTO dto = new ChangesetDTO();
dto.file = file;
dto.id = changeSetId;
if(context != null) {
dto.hasContext = true;
} else {
dto.hasContext = false;
}
List subNodes = node.selectNodes("*");
for(int j = 0; j < subNodes.size(); j++) {
Element subElement = (Element)subNodes.get(j);
String subElementName = subElement.getName();
if(!dto.operations.contains(subElementName)) {
dto.operations.add(subElementName);
}
}
changesets.add(dto);
}
}
}
</code></pre>
<p>As you may see, if this unit test fails (there is policy violation) the aplication will
not compile when building with maven.</p>
My Android cheatsheet2016-09-04T00:00:00+00:00https://xand.es/2016/09/04/my-android-cheatsheet<p>List of useful commands when working with Android Studio.</p>
<p>Global configuration:
<pre><code class="console">
$ export ANDROID_HOME="~/Android/Sdk"
</code></pre></p>
<pre><code class="console">
$ export PATH="$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/tools/bin:$ANDROID_HOME/platform-tools"
</code></pre>
<p>List available avds (Android Virtual Device):
<pre><code class="console">
$ avdmanager list avd
</code></pre></p>
My git cheatsheet2016-08-04T00:00:00+00:00https://xand.es/2016/08/04/my-git-cheatsheet<p>Merge branch without commit:
<pre><code class="console">
git merge branch-to-merge-with --no-commit --no-ff
</code></pre></p>
Managing differential backup with Proxmox2016-07-05T00:00:00+00:00https://xand.es/2016/07/05/proxmox-diff-backup<p>When working with <a href="https://www.proxmox.com/en/">Proxmox</a> the first question should be how are you going to manage backups.
It is clear that the responsibility of making machine internal backups is on the owner of the VM.
But, what happens if the Proxmox administrator (me, in this case) completely loses the server as a result of some kind of disk failure?
The answer is simple: do a backup!</p>
<h3>Backup schedule</h3>
<p>Proxmox offers an out-of-box solution for performing the task. However, the way it is by default is far from being perfect.
Basically, if using the solution as-is you will end up with a bunch of files where each of them will have approx. the size of the VM's disk.
However, there is a <a href="https://github.com/ayufan/pve-patches">solution</a> which offers differential backups. What are differential backups?
Long story short: on day 1 you make a full backup of your VM (<code>b1</code>), on day 2, you make another full backup of your VM (<code>b2</code>).
After that, you make a binary diff between <code>b2</code> and <code>b1</code> and store only this information (<code>d1</code>).
At any point of time if you have <code>b1</code> and <code>d1</code> you will be able to restore <code>b2</code>. However, you will need much less storage space compared to storing <code>b1</code> and <code>b2</code>.
The entire solution is built as Proxmox plugin and integrates perfectly into the GUI. Thanks, <a href="https://ayufan.eu/">ayufan</a> for this.</p>
<h4>Backup configuration</h4>
<p>After installing the plugin I spent some time trying to figure out the backup structure.</p>
<p><img src="/public/img/2016-07-05-proxmox-diff-backup/pve_backup_scr.png" alt="pve-backup-screenshot">
To illustrate the idea I made a screenshot of the backup view. If you configure it this way you will get backups for 24 days,
3 full backups plus 21 (7 days for each full backup) differential backups. Backups older than this will be automatically deleted.
The thing is, the differential backup will need only a fraction of storage space compared to full backup.</p>
<p>Let's see the numbers. In my case for a single VM a full backup needs about <code>4,3G</code>, so, if I want to store backups for 24 days you will need approx. <code>103,2G</code>.
At the same time, a differential backup needs only about <code>1,5G</code>, of course this amount depends on how much "movement" a VM gets during a day,
but here I take the largest value (normally it will go down to 500-600M). So, the full estimation would be <code>4,3G</code>x3 + <code>1,5G</code>x21 = <code>44,4G</code>. <code>103,2G</code> vs <code>44,4G</code>, not bad at all.</p>
<h3>Restoring the backup</h3>
<p>OK, so, you have your daily differential backup and obviously, you have it stored somewhere outside the machine itself. One day, I will make a post regarding this matter too.
Now, if bad things happen and you still with your host system available the restore may be done with a couple of clicks.
But what happens if your host system is no longer available? Well, you will need to restore the backup somewhere outside of your host machine.
To find a way to do this I had to investigate a little since it's not obvious at all.</p>
<h4>Case: Restoring a full backup</h4>
<p>If you need to restore a full backup file, the one with <code>vma.gz</code> extension the process is quite simple.
The file itself is gzipped vma container. You may read more about vma format <a href="https://pve.proxmox.com/wiki/VMA">here</a>.
The problem is, if you are restoring it outside the host, you will need to install the vma tool, which comes bundled with Proxmox itself.
Installing <code>vma</code> alone is not easy, however people from Proxmox forum found a <a href="https://forum.proxmox.com/threads/vma-archive-restore-outside-of-proxmox.14226/#post-76417">trick</a>,
which consists basically of installing all needed dependencies and then copying <code>vma</code> executable.</p>
<p>I surely need to prepare a docker image with this tool to avoid this installation process, meanwhile:</p>
<pre><code class="console">
# echo "deb http://download.proxmox.com/debian stretch pve" >> /etc/apt/sources.list
# wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg
# apt-get update
/* At this point you will probably need to check which package contains the required file */
# apt-get download pve-qemu-kvm
# dpkg --fsys-tarfile /tmp/pve-qemu-kvm_2.12.1-1_amd64.deb | tar xOf - ./usr/bin/vma > /usr/local/bin/vma
# chmod 755 /usr/local/bin/vma
</code></pre>
<p>At this point, if you try to execute <code>vma</code> tool the system will complain about some shared libraries. Let's install those dependencies:</p>
<pre><code class="console">
# apt-get install libglib2.0-0 libiscsi7 librbd1 libaio1 lzop glusterfs-common libjemalloc1
</code></pre>
<p>After this point you should be able to restore a full backup file. Let's do it.</p>
<p>Let's suppose we have a file <code>vzdump-qemu-158-2016_01_29-04_00_10.vma.gz</code>, the first step is to gunzip it.</p>
<pre><code class="console">
# gunzip vzdump-qemu-158-2016_01_29-04_00_10.vma.gz
</code></pre>
<p>After this operation we are going to have a <code>vzdump-qemu-158-2016_01_29-04_00_10.vma</code>, this is a <code>vma</code> container with our VM.
You may extract the information from here to see the contents, or, you may just deploy it to a working Proxmox host. Let's see what's inside:</p>
<pre><code class="console">
# vma extract vzdump-qemu-158-2016_01_29-04_00_10.vma -v data/
</code></pre>
<p>This command will extract the vma file into the <code>data</code> directory located at the same level. Actually, it will take some time to do it, be patient.
After the command completes you will end up with two files:</p>
<ul>
<li><code>qemu-server.conf</code>. This is the configuration of the VM.</li>
<li><code>disk-drive-scsi0.raw</code>. This is the raw contents of the disk. You may mount it to see the files.</li>
</ul>
<h4>Case: Restoring differential backup</h4>
<p>For restoring a differential backup we need one more additional step. Precisely we need first to merge <code>vcdiff</code> file, and then, perform the steps from the previous case.
For this task, we will need <code>pve-xdelta3</code> tool.</p>
<p><code>pve-xdelta3</code> tool is a slightly modified by ayufan version of <code>xdelta3</code> tool. You may read more about it <a href="https://github.com/ayufan/pve-xdelta3">here</a>,
also, you can download a deb package from <a href="https://github.com/ayufan/pve-xdelta3/releases">here</a> or compile from the source code.</p>
<p>For me the build process went smoothly on my Debian machine. I just downloaded a tarball, unpacked it and after executing
<code>./configure</code>, <code>make</code>, <code>sudo make install</code> I had <code>xdelta3</code> installed on my machine. For some reason the tool compiled from tarball is named <code>xdelta3</code>, while the tool installed with Proxmox plugin is named <code>pve-xdelta3</code>, for us it does not make any difference.</p>
<p>Let's say we have the following files: <code>vzdump-qemu-158-2016_01_29-04_00_10.vma.gz</code> and <code>vzdump-qemu-158-2016_01_29-04_00_10.vma.gz--differential-2016_01_31-04_00_07.vcdiff</code>.
For merging them, we need to execute the following command:</p>
<pre><code class="console">
$ pve-xdelta3 -q -d -R -s vzdump-qemu-158-2016_01_29-04_00_10.vma.gz \
vzdump-qemu-158-2016_01_29-04_00_10.vma.gz--differential-2016_01_31-04_00_07.vcdiff \
vzdump-qemu-158-2016_01_31-04_00_10.vma
</code></pre>
<p>When the command ends, you will have <code>vzdump-qemu-158-2016_01_31-04_00_10.vma</code> which is your full backup on that date,
you may go to the previous step to recover the information. Please note, that <code>xdelta3</code> already gunzipped the file.</p>
<h4>Mount the raw file</h4>
<p>In case you would like to mount the raw file you may use the following approach:</p>
<ul>
<li>Calculate the offset.</li>
<li>Mount the image as loop filesystem.</li>
</ul>
<p>Calculating the offset is straightforward.</p>
<pre><code class="console">
# fdisk -l disk-drive-scsi0.raw
Disk disk-drive-scsi0.raw: 50 GiB, 53687091200 bytes, 104857600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x2b23f370
Device Boot Start End Sectors Size Id Type
disk-drive-scsi0.raw1 * 2048 104857599 104855552 50G 83 Linux
</code></pre>
<p>With this command, you may see the sector size (line 3), which is 512 bytes. And you may see where the FS starts (line 10), which is on the sector 2048.
So, to calculate the offset you need to multiply 512b x 2048 = 1048576. This is where you FS starts. After getting this information you may mount:</p>
<pre><code class="console">
# mount -o ro,loop,offset=1048576 disk-drive-scsi0.raw /tmp/fs
</code></pre>
Assign IP address to docker container2016-06-12T00:00:00+00:00https://xand.es/2016/06/12/docker-with-known-ip<p>Many times when working with docker containers I feel the need of assigning a known beforehand IP address to a container. This is a huge advantage if you want to control the network access to and from a container with a tool like iptables. However, current docker version (1.11.1) does not allow this operation out of the box, but there is an official way of achieving this. Thanks to <code>docker network</code> command a user may create a fully customizable network and connect a container to it. You may find full information at the official Docker site, <a href="https://docs.docker.com/engine/userguide/networking/work-with-networks/">here</a>.</p>
<p>I will start with a clean docker installation on a test vagrant machine (Ubuntu Trusty). After installing docker, as usual, you may see <code>docker0</code> network interface. This a default bridging interface. I will follow the documentation and create an isolated network using the same subnet, addresses and names.</p>
<p>So the first step is to create a new network:</p>
<pre><code class="console">
$ docker network create -d bridge --subnet 172.25.0.0/16 isolated_nw
</code></pre>
<p>This network will allow me to use a <code>172.25.0.1</code> - <code>172.25.255.254</code> address range. If I run <code>ifconfig</code> now I will see that a new interface is created. In my case, docker calls it <code>br-98446a2a4f1f</code>. Just to be sure I reboot my machine to see if this network persists across reboots and it does.</p>
<p>Now I want to start <code>nginx</code> container with <code>172.25.0.2</code> address, I can do it with the following command:</p>
<pre><code class="console">
$ docker run --net=isolated_nw --ip=172.25.0.2 -d --name=my_nginx_01 nginx
</code></pre>
<p>If I get inside the container and run <code>ip addr</code> command I will see that the assigned IP address is, in fact, the requested one:</p>
<pre><code class="console">
root@597d1056bc32:/# ip addr
7: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:19:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.25.0.2/16 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe19:2/64 scope link
valid_lft forever preferred_lft forever
</code></pre>
<p>Now, I will start another container just to check the connectivity between them:</p>
<pre><code class="console">
$ docker run --net=isolated_nw --ip=172.25.0.3 -d --name=my_nginx_02 nginx
</code></pre>
<p>So, if I get inside a second container I'm able to perform ping and telnet with the first one:</p>
<pre><code class="console">
root@47f62e6951db:/# ping 172.25.0.2
PING 172.25.0.2 (172.25.0.2): 56 data bytes
64 bytes from 172.25.0.2: icmp_seq=0 ttl=64 time=0.253 ms
64 bytes from 172.25.0.2: icmp_seq=1 ttl=64 time=0.103 ms
64 bytes from 172.25.0.2: icmp_seq=2 ttl=64 time=0.140 ms
--- 172.25.0.2 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.103/0.165/0.253/0.064 ms
root@47f62e6951db:/# telnet 172.25.0.2 80
Trying 172.25.0.2...
Connected to 172.25.0.2.
Escape character is '^]'.
Connection closed by foreign host.
</code></pre>
<p>Time to check that linking between containers also works, I will start my containers this way:</p>
<pre><code class="console">
$ docker run --net=isolated_nw --ip=172.25.0.2 -d --name=my_nginx_01 nginx
$ docker run --net=isolated_nw --ip=172.25.0.3 --link my_nginx_01:my_nginx_01 -d --name=my_nginx_02 nginx
</code></pre>
<p>Then, if I connect to <code>my_nginx_02</code> container I will be able to ping and telnet <code>my_nginx_01</code> host.</p>
<pre><code class="console">
root@6e36ce623842:/# ping my_nginx_01
PING my_nginx_01 (172.25.0.2): 56 data bytes
64 bytes from 172.25.0.2: icmp_seq=0 ttl=64 time=0.161 ms
64 bytes from 172.25.0.2: icmp_seq=1 ttl=64 time=0.173 ms
--- my_nginx_01 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.161/0.167/0.173/0.000 ms
root@6e36ce623842:/# telnet my_nginx_01 80
Trying 172.25.0.2...
Connected to my_nginx_01.
Escape character is '^]'.
Connection closed by foreign host.
</code></pre>
<p>As you can see <code>my_nginx_01</code> resolves to the IP address assigned during the startup. With this configuration, you may be able to control your security perimeter using <code>FORWARD</code> chain in your <code>iptables</code> configuration.</p>
Configuring msmtp on Ubuntu2016-05-09T00:00:00+00:00https://xand.es/2016/05/09/configuring-msmtp<p>At the time being communication through email is less important than 10 years ago. Facebook, Whatsapp, Telegram, Slack, etc there is a feeling like old good email has less importance now for a meaning of communication. However, there is a field where this way of communication still being very important, when not the only way of communication. I'm talking about servers here.</p>
<p>For example, when ssh server detects break-in attempt it will try to send an email notification (or at least it could be configured in this way), or when some crontab-job fails, it also will try to send an email. But there is a problem, sometimes installing full-fledged MTA is counter-productive. Think about Raspberry Pi, if you install Postfix there it will be eating precious (and limited) resources of the machine itself. And we are not talking about the effort you will need to invest in an administration of the email server.</p>
<p>For myself, I found a much simpler solution called <a href="http://msmtp.sourceforge.net/">msmtp</a>. According to their website:</p>
<blockquote>
<p>msmtp is an SMTP client.</p>
<p>In the default mode, it transmits a mail to an SMTP server (for example at a free mail provider) which takes care of further delivery.
To use this program with your mail user agent (MUA), create a configuration file with your mail account(s) and tell your MUA to call msmtp instead of /usr/sbin/sendmail.</p>
</blockquote>
<p>In other words, you may connect to any SMTP server and send messages to any directions as if they were users on localhost. I will be using <a href="http://gmail.com">gmail.com</a> for this.</p>
<h3>Installation</h3>
<p>First of all, you will need to install it. Surprisingly for me, the version contained within Ubuntu repository is pretty old. However, you may build the latest version, this is not covered in this article.</p>
<pre><code class="console">
$ sudo apt-get update
$ sudo apt-get install msmtp msmtp-mta
</code></pre>
<p>Then I created a new email address at Gmail just to receive notifications from servers.</p>
<h3>Configuration</h3>
<p>Configuration is pretty straightforward.</p>
<pre><code class="plaintext">
# File: /etc/msmtprc
# Set default values for all following accounts.
defaults
auth on
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
logfile /var/log/msmtp.log
# Gmail
account gmail
host smtp.gmail.com
port 587
# If you're relaying through Gmail you can put here whatever you want. It's going to be ovewritten by your gmail's address.
from xandnotify@gmail.com
user xandnotify
# For me this password is not SO secret, but if you want to hide it there are many options.
password myverysecretpassword
# Set a default account
account default : gmail
aliases /etc/aliases
</code></pre>
<p>Please note the <code>/etc/aliases</code> file. We will need to use it later.</p>
<p>I'm not sure about the user which is going to execute <code>msmtp</code> command, so the log file should be created beforehand with proper permissions.</p>
<pre><code class="bash">
$ sudo touch /var/log/msmtp.log
$ sudo chmod 666 /var/log/msmtp.log
</code></pre>
<p>Now you can test the whole setup with the following command:</p>
<pre><code class="base">
$ echo "Hello there" | msmtp --debug your@email.com
</code></pre>
<p>If everything is OK you should receive an email at <code>your@email.com</code>. If not, try to see the debug output and figure out what went wrong.</p>
<h3>Fixing mail command</h3>
<p>For mail command to work you will need to put the following in <code>/etc/mail.rc</code>. Please note that in order to have <code>mail</code> utility you will need to install <code>mailutils</code> package which, unfortunately, on Ubuntu also includes Postfix server.</p>
<pre><code class="plaintext">
# File: /etc/mail.rc
set sendmail="/usr/bin/msmtp -t"
</code></pre>
<p>You can test that mail works with following command:</p>
<pre><code class="bash">
$ echo "This is a message body" | mail -s "Hello there from mail command line" xand@xand.es
</code></pre>
<h3>Fixing crontab</h3>
<p>When crontab fails to execute some command it sends an email message. In order to perform the operation, it uses <code>sendmail</code> executable. If you remember, in the beginning, I also installed <code>msmtp-mta</code> package. Doing so also creates a link <code>/usr/sbin/sendmail</code> to <code>/usr/bin/msmtp</code>. This way crontab does not need any additional configuration.</p>
<p>The problem here is that crontab always sends a message to a local user. I mean, if I log in as <code>admin</code> user and create a crontab job for this user the email will be sent to <code>admin</code> at localhost. It can be fixed using the following approaches.</p>
<h4>/etc/aliases</h4>
<p>This file defines email aliases for user. For example:</p>
<pre><code class="plaintext">
# File: /etc/aliases
root: admin@myserver.com, supervisor@myserver.com
default: catch-all@myserver.com
</code></pre>
<p>Given this configuration, all emails which go to <code>root@localhost</code> will also be sent to <code>admin@myserver.com</code> and <code>supervisor@myserver.com</code>. Also, all messages to unknown (non-existent) users will be sent to <code>catch-all@myserver.com</code>.</p>
<h4>MAILTO</h4>
<p>Crontab offers a more elegant way of sending emails. In the beginning of the crontab file, you can edit it with <code>crontab -e</code>, declare MAILTO variable. Like this:</p>
<pre><code class="plaintext">
MAILTO="xand@xand.es"
* * * * * /tmp/aaa.sh
</code></pre>
<p>That's it. Your comments are welcome.</p>