Microsoft Big Data– looking into the HDInsight Emulator

As outlined in a previous post Introducing Windows Azure HDInsight, HDInsight is a pay-as-you go solution for Hadoop-based big data processing running on Windows Azure. But you don’t even have to use Azure to develop and test your code. Microsoft has an HDInsight Emulator that runs locally on your machine and that simulates the HDInsight environment in Azure locally.

HDInsight emulator is installed through the Microsoft Web Platform Installer (WebPI) a 64-bit version of Windows (Windows 7 SP1, Windows Server 2008 R2 SP1, Windows 8 or Windows Server 2012). It also seems to install without issues on Windows 8.1.

After you have installed it you will see three shortcuts on your desktop: Hadoop command line, Hadoop Name Node status and Hadoop MapReduce Status. You should also see three folders created on your local disk (Hadoop, HadoopFeaturePackSetup and HadoopInstallFiles) as well as a number of new windows services being added. The HDInsight emulator is actually a single node HDInsight cluster including both name and data node and use local CPU for compute.

One of the sample Mapreduce jobs which is included is WordCount. To try it out we will first download the complete works of Shakespeare from project Gutenberg. The file is called pg100.txt and we copy it into the folder c:\hadoop\hadoop-1.1.0-SNAPSHOT – next just follow the steps as outlined in Run a word count MapReduce job. For most HDFS-file specific commands use hadoop fs –<command name>  where most of the commands are Linux system commands. For a full list of support Hadoop file system commands, type hadoop fs at the Hadoop command prompt, for a full explanation check out the Hadoop FS Shell Guide. The Hadoop command line shell can also run other Hadoop applications such Hive, Pig, HCatalog, Sqoop, Oozie,etc … To actually run the Wordcount sample job on the HDInsight emulator run:

hadoop jar hadoop-examples.jar wordcount /user/hdiuser/input /user/hdiuser/output

Once the job is submitted you can track its progress using MapReduce status webpage. When the job is finished, the results are typically stored in a fixed file named part_r_NNNNN where N is the file counter.

A second sample which is also included is estimating PI using Monte Carlo Simulation – check out this excellent video explaining the principle behind estimating PI using Monte Carlo Simulation. For an explanation of how to use this with an actual HDInsight cluster check out the Pi estimator Hadoop sample in HDInsight

hadoop jar hadoop-examples.jar pi "16","10000000"

References:

• Get started with the HDInsight Emulator
• Installing HDInsight emulator on your local machine (Video)
• The day HDInsight Died (again) – tips to help you when notice that the HDInsight emulator stopped working.

Tags van Technorati: Big data,Microsoft,HDInsight,Azure,Hadoop

Microsoft Big Data - Introducing Windows Azure HDInsight

Somebody once said - if you're going to stick around in this business, you have to have the ability to reinvent yourself, whether consciously or unconsciously.

This is the first in a series of blog posts about Big Data from a Microsoft perspective. I have always used my blog as a notebook as it helped me to get a clearer view on different topics. I hope that you stay with me in this journey in the exciting world of big data.

One of the challenges with working with big data is that the volume and the expected growth of data volume can be quite hard to predict. When starting with Big Data a cloud platform is an ideal way to start given its pay per use model and the flexible scalability model. Another thing to consider is the fact that Big Data technology evolves quite rapidly and cloud providers such as Microsoft will evolve along giving you the opportunity to work with the latest technology. So if you are just getting started and you have a Microsoft background Windows Azure HDInsight might be a good place to start. Also remember that if you have an MSDN account you are eligible for Azure monthly credits up to 150 USD.

Microsoft worked together with Hortonworks to build their Hadoop-based big data solution, the Hortonworks Data Platform (HDP). It exists in 3 different flavors:

• HDInsight is an Apache Hadoop-based distribution running in the cloud on Azure. Apache Hadoop is an open source framework that supports data-intensive distributed applications. It uses HDFS storage to enable applications to work with 1000s of nodes and petabytes of data using a scale-out model.
• Hortonworks Data Platform( HDP) for Windows is a complete installation package which can be installed on Windows Servers running on premise or on virtual machines running in the cloud.
• Microsoft Analytics Platform System (formerly called Microsoft PDW)

In this post we will focus on Windows Azure HDInsight. HDInsight 3.1 which contains HDP 2.1 and is currently the default version for new Hadoop clusters (Always check Microsoft HDInsight release notes for the latest version info). At the core HDInsight is providing the HDFS/MapReduce software framework but related projects such as Pig, Hive, Oozie, Sqoop and Mahout are also included. 

HDFS (Hadoop File System) is a distributed file system designed to run on commodity hardware and is highly fault tolerant by nature. HDFS was developed by Doug Cutting and Mike Cafarella when they worked at Yahoo on the Nutch search project in 2005 and was inspired by the Google GFS white paper (See an interview with Doug Cutting, the founder of Hadoop (April 2014) and How Yahoo spawned Hadoop, the future of Big Data). In Hadoop, a cluster of servers stores the data using HDFS, each node in the cluster is a data node and contains a HDFS data store and execution engine. The cluster is managed by a server called the name node.

This distributed file system however poses some challenges for the processing of data and this is where the MapReduce paradigm  comes in which was also inspired by Google (MapReduce: Simplified Data Processing on Large Clusters, 2004). The term itself refers to the two basic computations in distributed computing, map (determining where the data is located in the different nodes and moving the work to these nodes) and reduce (bringing the intermediate results back together and computing them). These Mapreduce functions are typically written in Java, but you can use Apache streaming to plug in other languages

There are a number of advantages of using HDInsight:

• You can quickly spin up a Hadoop cluster using the Azure Portal or using Windows PowerShell
• You only pay for what you use. When your Hadoop processing jobs are complete, you can deprovision the the cluster and retain the data because Azure HDInsight uses the Azure Blob storage as the default file system which allows you to store data outside of the HDFS clusters.
• Microsoft provides deep integration into the rest of their BI stack such as PowerPivot, Powerview, Excel, etc. …
• HDInsight is exposed using familiar interfaces for Microsoft developers such as a .NET SDK (see for example Submit Hive jobs using HDInsight .NET SDK) and PowerShell

That’s it for now – in a next post I will delve a little deeper in how you can setup your first HDInsight cluster, the different architecture components and I will look into the HDInsight emulator.

References:

• Windows Azure HDInsight
• HDFS Architecture Guide
• Introduction to Hadoop in HDInsight
• Microsoft Big Data
• Hortonworks Data Platform

Tags van Technorati: Big data,Microsoft,HDInsight,Azure,Hadoop

Driving sustainable user adoption for SharePoint and Yammer – Part I

A while ago I did a presentation at the Future of Business is sharing event on user adoption of collaboration technologies within companies. A good user adoption strategy – this is not the same as change management – is key in getting a collaborative environment accepted within a company.

Although SharePoint has been hugely successful in the past few years – there is still a gap in satisfaction between IT pros and business managers – SharePoint met the expectations of  73% of the former, and of 62% of the latter (Source: Microsoft SharePoint faces tough future, Forrester says). In my opinion, this is because in most organizations, user adoption is just an afterthought. People often confuse user adoption with training, so a typical reaction is – “let’s send our end users to some training” – probably a technical training about site collections, versioning, web parts, etc… and what happens 5 weeks after the training they fall back in the old way of working.  Training is important, it allows people to make the first jump, but it has to be contextual, if you have build a solution around managing projects with SharePoint, make a specific training on the benefits that people will get when using SharePoint.

But user adoption really is about people getting to know your solution, understand it and use it in a correct manner. In the end the success of a deployment such as Office 365/Yammer or SharePoint Online is measured by sustained user adoption. Why the emphasis on “sustained”?  A study by the Altimeter Group about usage of social computing tools found that after an initial spike in enthusiasm and usage, you typically see a gradual decline in usage until only limited groups within your company are you using the solution. [Altimeter Group– making the business case for Enterprise Social Networks] – this applies to collaboration tools in general.

Getting beyond the early adopters of a technology solution, is not easy and in a perfect world – with only IT consultants (just kidding :-)) - everything would work magically. In a real world “Build and they will come” simply does not work. one of the main reasons is that people are fundamentally resistant to change, so you will need to put some effort in it to explain them why they need to change. If a new idea or way of work is initiated and self-sustaining, it will only survive if it gets adopted by a critical mass of users (typically you will need to at least >50% adoption)

The two most important things to focus on are Why and What?  Why do you need SharePoint (or Yammer, Yambla,…) – and what are the business problems you are going to solve or mitigate.  People know about file shares and they work with on a daily basis and then comes along this  this new product – SharePoint – but untill you can explain them how they need to use in their daily work and routines, they will not adopt it. Social, Yammer – why? What is the added value of using the SharePoint newsfeed or Yammer groups to someone in accounting – don’t push a certain feature if you can’t answer why people would need it and how they can use it.

It is interesting to see that even after you have deployed a file sharing and collaboration solution, people still send out e-mails with attachments instead of a link to the file. There are two killer applications in an enterprise, Excel and e-mail. And even with companies advocating zero-email (but struggling to actually make it happen), I don’t see e-mail disappearing anytime soon. So instead of banning it, embrace it and integrate it into your solution and use it in the more efficient way.
In a next post I will talk about how to leverage a user adoption team to make your collaboration platform deployment a success.
“Success starts with deployment, it does not end with deployment” – a it is necessary but not sufficient.

Future of business is sharing - IMEC Share, innovate, collaborate and excel

IMEC Share - Innovate, collaborate and excel from Joris Poelmans

Ten indispensable tools for SharePoint 2013 developers

1. CamlDesigner 2013 – provides you with a graphical inferface which allows you to   build CAML queries for single lists as well as queries that can be executed with SPSiteDataQuery. You can also get code snippets for the server-side object model, the .NET client-side object model, the JavaScript client-side object model and REST. This tool has been developed by Karine Bosch and Andy Van Steenbergen – two of the board members of BIWUG (Belgian SharePoint User group) 
   
2. SharePoint Manager 2013 – is a SharePoint object model explorer. It enables you to browse every site on the local farm and view every property.
   
3. ULSViewer  - there are other tools out there – check out SharePoint ULS log viewer tool comparison and verdict
4. SharePoint 2013 Client Browser – requires no installation –simply unzip the exe – allows you to explore the SharePoint object hierarchy.
5. Fiddler – is a web debugging tool which allows you to investigate all HTTP traffic (REST calls and XML or JSON responses). It also has some built in features to profile app performance and spot bottlenecks – also check out Fiddler PowerToy – Part 2 : HTTP performance
6. SharePoint 2013 Search Query Tool v2  - can be used to query, test and debug search queries – for both SharePoint 2013 on-premise and SharePoint online search queries. I also use it for examing and tuning ranking of search results. See understanding item ranking in SharePoint 2013 search for more details
7. SPCop Community Edition – this is a Visual Studio extesion which analyzes your SharePoint code which was created by Matthias Einig and which is based on the SharePoint Code Analysis Framework
8. SPFastDeploy – Visual Studio extension which allows to push individual files for deployment in SharePoint apps without requiring you to do a full re-deploy every time there is a change. It makes developers a lot more productive using the SharePoint 2013 hosted app model. Excellent tool built by Steve Curran.
9. SharePoint Color Palette Tool – the new SharePoint 2013 theme model (also called composable looks) allows you to brand your SharePoint 2013 environment in a new way. One of the key components of a composable look is a .spcolor file which defines the color elements. The color palette tool is  a free utility that enables you to develop spcolor files interactively
10. REST client plugin for Google Chrome – Excellent tool for creating REST requests – some developers might however favour the REST Postman plugin for Google Chrome.

Technorati Tags: SharePoint,Codeplex,Search,CAML,SP2013,Office365

Understanding item ranking in SharePoint 2013 search

SharePoint 2013 allows you to view the result of the ranking model by retrieving the rankdetail managed property. The rankdetail property will only be returned when there are less than 100 results in the returned search result set.

There is however an interesting tool hidden inside SharePoint which outlines the rankdetails calculation which I found in Explain rank in SharePoint 2013 search. SharePoint 2013 contains a built-in  application page called /_layouts/15/explainrank.aspx which accepts two mandatory parameters:

• q – which contains the query
• d – which specifies the path of the item for which you want to see the rankdetail

You can download a specific Explainrank search display template which will incorporate a link to the explainrank.aspx page with the required parameters. An alternative for this which will also work on Office 365/SharePoint Online is the Mavention Search Ranking app. My current favorite for the moment is the SharePoint 2013 Search Query Tool which also allows you to show rankdetails and rank calculation.

For a deep dive explanation of all the fine details of search ranking you should definitely take a look at customizing ranking models to improve search relevance in SharePoint 2013. On a very high level relevance is determined by two different types of parameters:

• Dynamic ranking: ranking parameters linked to the search term which is being used
• Static ranking: ranking parameters which are being applied independent of the search query which is being used. When you look at the standard ranking algorithm being used when doing a content search – the following static parameters are taken into account:
  • Clickdistance : number of clicks between an authoritative page and items in the index.
  • QueryLogClicks, QueryLogSkips and LastClicks : use click through behavior to see if results are considered relevant by users
  • EventRate : activity tracking of usage events (clicks or view) – items with high usage activity get a higher activity rank score than less popular items. This is activity on items in SharePoint outside of the search pages.
  • URLDepth : documents which have a longer URL are considered to be less relevant
  • InternalFileType – SharePoint 2013 prioritizes some files differently based on the file type – this is the current ranking (PPT,DOC,HTML,ListItem,Image,Message, XLS,TXT,XML). The most signifcant difference with 2010 is the fact that PowerPoint get a relatively higher weight in 2013 and Excel lower.
  • Language – some languages seem to be favoured – I’m still looking into the details of this

One of the surprises with SharePoint 2013 search is that it does not take into account the freshness of results – Mikael Svenson wrote an interesting post about this in  Adding freshness boost to SharePoint Online and this functionality is now also available in the SharePoint 2013 Search Query Tool

References:

• Understanding item ranking in SharePoint 2013 search
• Customizing ranking models to improve search relevance in SharePoint 2013
• Explain rank in SharePoint 2013 search
• Changing the ranking of search results in SharePoint 2013
• Overview of analytics processing in SharePoint 2013
• How SharePoint 2013 ranking models work
• SharePoint 2013 Search ranking and relevancy Part 1: Let’s compare to FS14

The end of SharePoint autohosted apps

Up until a couple of weeks ago you basically had 3 options to develop a SharePoint app as shown in the figure below – one of the options – Autohosted apps was however available in preview mode and therefore not recommended for production use. Mid May Microsoft pulled the plug out of this one – see Update on Autohosted Apps Preview Program

As outlined by Andrew - Update on Autohosted Apps Preview Program – this is probably a good thing because although the deployment model was quite flexible you were never sure what were the actual limitations with regards to allocated CPU time,data out,storage and memory usage. So for now, we have to wait and see what enhancements become available for provider hosted SharePoint apps to make it as flexible in deployment as autohosted apps but while still providing you with sufficient control and troubleshooting capabilities.

References:

• SharePoint Autohosted Apps are not more – deprecated, discontinued … dead
• How to: Convert an autohosted app for SharePoint to a provider-hosted app
• Update on Autohosted Apps Preview Program

Technorati Tags: SharePoint,SP2013,Apps,Azure,Microsoft,Development

The future of business process apps – a Microsoft perspective

The future of business process apps - a Microsoft perspective from Joris Poelmans

Technorati Tags: K2,SharePoint,BPM,Process+Management,Mindflow,Mobile,Organizational+theory

K2 FastFWD Event in Gent on May 21st

 

I will be speaking at the K2 FastFWD event in Gent on May 21th. Don’t forget to register on K2 FastFwd Global Roadshow (Gent, Belgium).

Join K2 for our exclusive K2 for SharePoint launch event at the prestigious Ghen t Marriot Hotel in Belgium. Earlier this year K2 released K2 for SharePoint 2013, allowing users to build and run business applications within SharePoint - using forms, workflow, data and reports. This announcement was made in conjunction with the K2 Appit launch at the K2 User Conference in Las Vegas. With this successful launch, we are now bringing Vegas to you!

We have an exciting agenda planned, and are delighted to announce that Principal Consultant from RealDolmen, Joris Poelmans, will be giving insight into how companies are using SharePoint today.

Learn how to start solving your business challenges by delivering meaningful business applications with K2 Forms, Workflows and Data. Fast.

AGENDA
09:00 - 09:30 - REGISTRATION AND BREAKFAST

09:30 - 09:50 - WELCOME AND VISION - FAST FWD
10:00 - 11:00 - FWD THINKER KEYNOTE ON BUSINESS PROCESS APPS
Gain thought leadership perspectives from respected FWD thinker and Principal Consultant at RealDolmen, Joris Poelmans, about the future of business process applications.
11:00 - 11:15 - BREAK

11:15 - 12:15 - K2 FOR SHAREPOINT DEMO - APPS THAT SPAN YOUR LOB SYSTEMS
Protect your investment in SharePoint and other systems by connecting them through a no-code, cloud-based platform, creating workflow and forms apps that seamlessly integrate with SharePoint online, on-premises or both..

12:15 - 12:45 - WHATS NEW IN K2?
Get a first-hand preview of the latest K2 releases:

• Appit
• Mobile
• Office 365
• SharePoint online
• Social
• InfoPath alternatives - K2 smartforms

12:45 - 13:00 - Q&A PANEL

13:00 - 15:00 - NETWORKING & LUNCH

WHY ATTEND?
K2 for SharePoint 2013 allows users to:

• create workflows that span lists and libraries in different SharePoint sites
• build web and mobile apps that use SharePoint data; and
• build components, like forms and workflows once and reuse them across all your applications – in and out of SharePoint.

This launch event allows you to network and hear from leaders in organisations with similar challenges to yours. K2 customers now have the ability to create scalable and secure no-code apps that span on-premises, cloud and hybrid.

 

Technorati Tags: K2,Event,SharePoint,Workflow,SharePoint2013,Forms

Slidedeck Yammer Social Data Mining (SharePoint Saturday Belgium 2014)

The slidedeck from my session on SharePoint Saturday Belgium 2014 is now available on slideshare.

Technorati Tags: Yammer,Data+mining,REST+api,PowerPivot,PowerBI,Office365

BIWUG on authenticating SharePoint apps and supporting SharePoint in the Cloud

On the 20th of May BIWUG is organizing the final session before summer holidays – don’t forget to register for BIWUG2005 because there are some great sessions planned.

Authenticating client-side applications in cross-domains & hybrid environments (Speaker: Stephane Eyskens, SharePoint Technical Architect - http://www.silver-it.com/ )

SharePoint 2013 comes with many new challenges among which the rising of client side technologies. In this session, I will not talk about Knockout, AngularJS whose everybody is talking about, I'm rather going to talk about diffent techniques to tackle client side related challenges, particularly regarding cross-domain & authentication by leveraging HTML5, CORS, infrastructure workarounds and SharePoint built-in APIs. I'll be talking about two different real world implementations (1 project 100% on-prem and 1 onprem+azure) and two different ways of solving these issues. After this session, developers should have a better understanding of the tools & techniques available to handle those complex problems.

Supporting SharePoint in the Cloud: insights, figures, challenges,… (Speaker: Pascal Benois, EMEA Technical Support Lead Office 365 at Microsoft)

What does it take to support Office 365 for 2.500.000 customers in EMEA ? What are the biggest challenges, most common issues, …?  With the success of O365, the IT Pro and Dev landscapes are evolving and raising new challenges. One thing is for sure, there will be no way back. During this session, I will expose what the current situation is and what it is expected to be in the near future, how Microsoft address the partners and “broad commercial” ecosystems in terms of support and delivery. Also expect some insights about upcoming features and roadmap.

 

Technorati Tags: Community,biwug,sharepoint,Office365

Event Future of Business is Sharing

On  May 16th RealDolmen and Kluwer are organizing the “Future of Business is Sharing” event with a number of interesting sessions.

Kennis is macht, kennis delen is machtiger. Samenwerken bepaalt het succes van mensen en bedrijven. Het stelt hen in staat die extra stap vooruit te zetten die het verschil kan maken. In ons persoonlijk leven doen we niets anders: denk aan het succes van YouTube, Naai- en Repair Cafés waarop informatie wordt gedeeld en tips & tricks worden uitgewisseld.

Maar vertaalt dit zich ook in onze professionele wereld? Het lijkt alsof bedrijven achterblijven. Medewerkers zitten vaak vast in de kokers van hun jobgrenzen, informatie in hoofden van individuen. Disciplines staan al te vaak naast elkaar en mensen gevangen in één specifieke rol.

Tijdens dit event slaan we met een aantal betrokken partijen de handen in elkaar. We staan met u stil bij het belang van kennis delen en samenwerken over domein- en  bedrijfsgrenzen heen. 

• Samen voor ons eigen (Speaker: Dirk Draulans) - De transformatie was een sluipend proces, waarbij samenwerking over de groepsgrenzen heen getild werd. Vanuit een evolutionair perspectief is onze aanleg tot samenwerking echter niet vanzelfsprekend en lijkt het regelrecht in te gaan tegen de natuurlijke selectietheorie van Charles Darwin…
• Social learning: voorbij de hype (Speaker: Isabel De Clercq) - Over “the rise and fall” van sociale netwerken in organisaties. Over de onzin maar vooral de zin van social learning. Over het stimuleren van krachtige kennisdeling in uw organisatie. Dit is een weg-van-de-hype-sessie: verwacht u aan eerlijke verhalen en bruikbare tips & tricks. 
• imec Share: Innovate, collaborate and excel (Speakers: Francis Bruynseels & Joris Poelmans) -  Imec verricht onderzoek dat tot de wereldtop behoort in het domein van nano-elektronica. Tijdens deze sessie staan we stil bij de visie van imec rond samenwerking - zowel binnen de organisatie als met externe partners wereldwijd -, de uitdagingen en de transitie die de organisatie doormaakt.
  Een concreet praktijkverhaal waarin visie, change en technologie aan bod komen.
• Paradigm shift: niet meer en vooral niets minder (Speaker: Hugo Der Kinderen) -Samenwerken over alle grenzen heen en kennisdelen. Het klinkt als muziek in de oren. Maar zal het ook lukken in uw organisatie? Intense kennisdeling lukt enkel in bepaalde organisatiemodellen. Hoe ga je van een werkplek waar 'geld in ruil voor arbeid' centraal staat, naar een plek waar 'ontwikkeling, kennisdeling en synergie' overheersen.

To register or for more information check out thefutureofbusinessissharing.com 

Technorati Tags: OHNW,Collaboration,Event,Innovation

Power BI, Power View, Power Query and PowerPivot – interesting links

 

• Microsoft Virtual Academy – Faster insights to Data With Power BI Jump Start
• Getting started with Microsoft Power BI For Office 365 (Video duration 00:01:13)
• Publish a workbook to a Power BI site and prepare it for Q&A
• Microsoft adds natural language search over data to Office 365
• Create a Data Model in Excel
• Power Query 101
• Introduction to the Data Model and Relationships in Excel 2013
• Power Query Basics – Indeed! (Part 1) and Power Query Basics – Indeed (Part 2) and Part 3 – Power Query API – Table as parameter for a query Indeed! and Part 4 – Power Query API – Accumulating Data Indeed! and Part 5- Power Query API – Refreshing Data Indeed!
• Sharing and discovering queries using Power Query and Power BI
• Tutorial: PivotTable data analysis using a Data Model in Excel 2013
• Guide to the Power Query Ribbon
• Analyzing Data with Power BI from beginning to end

Technorati Tags: Power+BI,PowerView,PowerPivot,Q&A,Business+Intelligence

Solving invalid pointer array error message in Power View for Excel 2013

Last week my Power View reports suddenly gave an error “We couldn’t get data from the Data Model. Here’s the error message we got: Invalid pointer array”

Luckily a colleague pointed to the correct solution on the forums – KB2752087 breaks PowerView in Microsoft Excel 2013. Apparently this was caused by the automatic update KB2752087 – so uninstalling this update would solve the above error but since it is pushed out by Windows Update that will not help a lot. The solution for this seems to be installing hotfix KB 2837666

Technorati Tags: Excel,Excel2013,PowerBI,PowerView,Power+BI

Managed metadata – lookup terms using custom properties in CSOM

 

SharePoint Server 2013 introduces the notion of custom properties on specific terms in managed metadata termset and with  the added support for .NET client object model (CSOM) for managed metadata APIs you have some quite interesting scenarios that you can imagine (Check out managed metadata and navigation in SharePoint Server 2013)

A typical scenario is one in which you have a master system for which you want to export some parts to a termset in the managed metadata store. To make this to happen you will however need a key from the master system to be linked to the different terms that you are creating. Listed below is some sample code to look for a specific term based on a custom property.

   1:          static Term GetTermByCustomID(string customid)

   2:          {

   3:              Term retTerm = null;

   4:   

   5:              CustomPropertyMatchInformation matchinfo = new CustomPropertyMatchInformation(Program.clientContext);

   6:              matchinfo.CustomPropertyName = "ID";

   7:              matchinfo.CustomPropertyValue = customid;

   8:              matchinfo.TrimUnavailable = false;

   9:   

  10:              var terms = termSet.GetTermsWithCustomProperty(matchinfo);

  11:              Program.clientContext.Load(terms);

  12:              Program.clientContext.ExecuteQuery();

  13:   

  14:              if (terms.Count > 0)

  15:              {

  16:                  retTerm = terms[0];

  17:              }

  18:              

  19:              return retTerm;

  20:   

  21:          }

One of the things which puzzled me for quite a while was the TrimUnavailable property – but recently the documentation got updated which made it a little more clear - see TermSet.GetTermsWithCustomProperty method (String, String, StringMatchOption, Int32, Boolean) – apparently it is indicates whether to trim out Term objects that have the IsAvailableForTagging property set to false.

Technorati Tags: SharePoint,CSOM,managed+metadata,termsets

Session Yammer and SharePoint social data mining at SharePoint Saturday Belgium 2014

I will be doing a session on social data mining on SharePoint Saturday Belgium 2014 –  Yammer and SharePoint social data mining - actionable insights into what drives your co-workers and employees

Enterprise social tools such as Yammer and SharePoint allow you to listen to what is driving your employees. In this session we will examine how analytics tools can be use to monitor employee sentiment, predict employee churn or how they can be incorporated in an employee evaluation process. Attendees will explore how to use the different social APIs - both for Yammer and SharePoint as well as data mining and analysis tools to present employee social data in dashboards using Excel Services, Power BI and Visio Services which provide actionable insights.

Also check out the other speakers on SharePoint Saturday Belgium 2014 and the different sessions at SharePoint Saturday Belgium 2014

Technorati Tags: BIWUG,SharePoint,Yammer,Data+mining,Insights,PowerBI,Office365

Search refiners in SharePoint 2013 and 2010: differences and some tips

An out of the box SharePoint Server 2010 installation shows six refiners or “Categories” including  Result Type, Site, Author, Modified Date, Managed Metadata columns, and Tags in the refinement panel webpart on the content search page.

The refinement panel webpart provides a summary of search results and enables users to filter results (See Refinement Panel Overview)

Figure Search refiners in SharePoint 2010

Refiners (SP2010)	Managed Property
ResultType	FileExtension
Site	SiteName
Author	Author
Modified Date	Write
Managed Metadata Columns	ows_MetadataFacetInfo
Tags	ows_MetadataFacetInfo, popularSocialTags

In SharePoint Server 2013 you will get a different set of default refiners  File Type, contentClass, ContentTypeId, WebTemplate, DisplayAuthor and LastModfiedTime. The way that these search refiners are being rendered is determined by the display template which is selected.  This makes it a lot easier to modify the refinement panel. You will find the different search refiner display templates which are being used in the master page gallery underneath /Display Templates/Filters/ (The full URL will be something like http://yoursharepoint/_catalogs/masterpage/display templates/filters/)

If you look for example at the standard Refinement Item display template, you will also see why why the contentClass, ContentTypeId and WebTemplate are not shown in the search refinement panel.

A common question that you might get when upgrading from SharePoint 2010 to SharePoint 2013 is how to add the Tags refiner back in the refinement panel. This is actually pretty simple, just go to the refiners preview panel and add in the same managed property ows_MetadataFacetInfo.
One of the changes in SharePoint Server 2013 is the fact that refiner counts are not shown anymore by default. Fortunately it is actually quite easy to change this again. You can modify the display template of the refiners – the only thing you have to change is, setting the ShowCounts value to true.

The final result should look like this. If you need to repeat this step on multiple search centers you can use the PowerShell enable refiner count in SharePoint 2013

References:

• How to add refiners to your search results pages in SharePoint 2013
• Configure refiners and faceted navigation in SharePoint Server 2013 [Technet]
• Query refinement in SharePoint 2013 [MSDN]

Why enterprise social networks matter

In an time where globalization, co-creation and cross-company cooperation is essential for a company’s survival it is quite strange to see why companies still hesitate to embrace enterprise social networks (See Trends impacting collaborative tools and platforms for more background information)

The fact that we are more and more working in different locations makes it more difficult to use our traditional means of communication and collaboration. Especially in organization where physical face-to-face meetings are less and less the norm you see a change in how people collaborate.

In the late 1970s Tom Allen (MIT) undertook a project to determine how the distance between engineers’ offices was related to the way that they communicated and collaborated. One of the key findings was the fact that the probability of communicating at least once a week drops significantly if people are more than 50 feet apart.

We also see that people are not only working on multiple locations (separation by space) but also more and more asynchronously. This might be caused by the fact that people work in different time zones but also by the rise of flex-working.

This separation in both distance and time provides for some specific challenges which are hard to resolve using traditional technologies. Typical intranet and email solutions are broadcasting and are push technologies. Especially with email you see that the sender chooses the recipients (push medium instead of pull) and replies overload mailboxes.

Another side effect of using email as a many-to-many async mechanism is that knowledge gets trapped in email clients with no easy means of discovery and searching for knowledge. When a new employee enters the company he will probably be subscribed to a large number of email distribution lists but he has no access to all of the conversations which were ongoing or which happened in the past.
It is however essential that to create an organization in which social is embedded, that you need something more than enabling software tools. It is critical to have a social computing strategy that aligns business objectives, required cultural change, supporting processes and technology.

Whether you will be using SocialCast (VMWare), Salesforce Chatter, Tibco Tibbr, Jive or SocialText – a succesful implementation implementation all boils down to these 5 critical components:

• Clear executive sponsorship and cultural willingness of the organization. If management is not listening or does not show it is listening, change will not happen.
• Explicitly define specific communities of users to target as well specific use cases.
• Define desired results and/or objectives for the different use cases and continuously measure the output. Make sure that the output is aligned with the core key performance indicators (KPIs)
• Use an iterative approach – plan/experiment/learn – but make sure that the different learning experiences and use cases are aligned with your global social strategy.
• Identify change agents (or champions or key users) who will get the social network started

References:

• Reimagining with Social (Microsoft)
• Working social: onboarding (Yammer)
• Channel 9 video -  Driving business strategy with Enterprise Social – a large part of this blog post was inspired by this interesting session at SharePoint Conference 2012
• 5+8+3 step to successfully launch an enterprise social network? Strategy

Images in SharePoint Server 2013 search

You might have noticed that SharePoint Server 2013 does not crawl image files (jpg, png,gif,…) and that it only returns these images as list items. Steve Mann wrote a series of blog posts about this (see links in the references) section – in the next steps I will outline a summary of the required steps as reminder:

1. Add image to file types to search and run a full crawl of all content
2. The image result type still uses the Default item display template – create a copy of the image result type and modify it to use the Picture item display template as outlined in SharePoint 2013: Image Preview in Search Results - Part II, strangely enough this will work different for images stored in a picture library or in another library – so you still have to modify the Picture item display template
3. A last possible step is that you modify the hover panel so that it will also include a bigger preview of the image – as outlined in SharePoint 2013: Image Preview in Search Results - Part III

References:

• SharePoint 2013: Image Preview in Search Results – Part I
• SharePoint 2013: Image Preview in Search Results - Part II
• SharePoint 2013: Image Preview in Search Results - Part III

Using folders in SharePoint document libraries – some guidance and tips

The discussion about whether folders in SharePoint are a good or a bad idea comes up quite often – I already wrote a number of posts about this topic: Follow up – Folders in SharePoint document libraries – why??(October 2008)  as well as Folders and metadata – again (February 2010).

After some years of working with SharePoint and talking to a lot of users I have come to the following conclusions and I have re-written the original blog post to take into account the different enhancements of SharePoint 2013.

Sometimes users simply want to group documents in small groups since they are used to work this way, these groups of documents can be ad hoc and don’t necessarily translate into document metadata. So declaring folders in SharePoint to be evil is just one bridge too far – sometimes they have their use. But if possible and when the added value is clear you should indeed translate your folders into metadata especially when the naming of the folders contains a specific meaning.

Advantages of using folders in SharePoint document libraries:

• It looks familiar to people used to work with file shares
• Folders are required when you have a large number of documents in a document library – if you look at the Software boundaries and limits for SharePoint 2013 guidance on Technet – you will notice that the supported limit is 30 million documents in a single library but it also states that can use nested folders to store this amount of documents.
• It is possible to define security on folders
• Possible to define metadata on folders. Remark: the metadata is not replicated on the documents within the folder, if you want to do push down metadata from a folder level to the documents contained within you can use common default values or location-based metadata defaults (available in SharePoint 2010 and 2013) or document sets.
• Possible to create an alert on a folder
• SkyDrive Pro and Explorer View don’t really support metadata, by combining folders together with metadata you can still provide for some classification support.
• If you really want to use folders, you can still create a “without folders” view in SharePoint.

Disadvantages of using folders in SharePoint document libraries:

• Not possible to use it for filtering or to create views
• Only allows for adding a single dimension of information, it also requires you to drill down into sub folders before you find the documents. All in all it is quite a rigid structure which leads to difficulty in finding content. It is especially annoying since you don’t see a count for the number of items contained in a folder so you click all the way down just to find that there is no document stored in the folder.
• When you move a document to another folder (because you miscategorized it) - the url will change. This is not the case when you change a metadata field
• Will increase the length of the URL - remember URL length for Docs in SharePoint is still at 260 characters
• Contributors can change your carefully architected folder structure containing your classification.

Just keep in mind folders should be the exception in SharePoint and not the rule. Do not hesitate to leave a comment or your opinion.

Technorati Tags: SharePoint,Folders,Metadata,SharePoint+2013